[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520271089 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003207] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008314] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014072] *** VALIDATE selinux *** [ 0.021685] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026436] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029095] *** VALIDATE tmpfs *** [ 0.030426] *** VALIDATE proc *** [ 0.032159] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.035117] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042388] debug: unmapping init [mem 0xffffffffa3859000-0xffffffffa3860fff] [ 0.044262] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050010] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053263] rcu: Hierarchical SRCU implementation. [ 0.055438] smp: Bringing up secondary CPUs ... [ 0.056559] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.061133] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132228] node 0 deferred pages initialised in 67ms [ 0.140033] devtmpfs: initialized [ 0.141300] x86/mm: Memory block size: 128MB [ 0.145283] gcov: version magic: 0x41383552 [ 0.147176] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155268] pinctrl core: initialized pinctrl subsystem [ 0.157288] [ 0.157966] ************************************************************* [ 0.162012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164068] ** ** [ 0.167009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170012] ** ** [ 0.173009] ** This means that this kernel is built to expose internal ** [ 0.175045] ** IOMMU data structures, which may compromise security on ** [ 0.178011] ** your system. ** [ 0.180009] ** ** [ 0.182010] ** If you see this message and you are not debugging the ** [ 0.184008] ** kernel, report this immediately to your vendor! ** [ 0.186007] ** ** [ 0.188009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190011] ************************************************************* [ 0.193566] NET: Registered protocol family 16 [ 0.196392] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206117] cpuidle: using governor menu [ 0.207819] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210637] PCI: Using configuration type 1 for base access [ 0.212252] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.220205] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.221058] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225079] cryptd: max_cpu_qlen set to 1000 [ 0.226258] ACPI: Added _OSI(Module Device) [ 0.227000] ACPI: Added _OSI(Processor Device) [ 0.229013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235470] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243741] ACPI: Interpreter enabled [ 0.245050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246010] ACPI: Using IOAPIC for interrupt routing [ 0.248111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261314] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.270068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.276120] acpiphp: Slot [2] registered [ 0.277111] acpiphp: Slot [5] registered [ 0.279103] acpiphp: Slot [6] registered [ 0.280126] acpiphp: Slot [7] registered [ 0.282166] acpiphp: Slot [8] registered [ 0.283101] acpiphp: Slot [9] registered [ 0.285206] acpiphp: Slot [10] registered [ 0.286078] acpiphp: Slot [3] registered [ 0.288056] acpiphp: Slot [4] registered [ 0.289087] acpiphp: Slot [11] registered [ 0.290057] acpiphp: Slot [12] registered [ 0.292084] acpiphp: Slot [13] registered [ 0.293193] acpiphp: Slot [14] registered [ 0.295080] acpiphp: Slot [15] registered [ 0.297414] acpiphp: Slot [16] registered [ 0.300106] acpiphp: Slot [17] registered [ 0.303113] acpiphp: Slot [18] registered [ 0.304151] acpiphp: Slot [19] registered [ 0.306119] acpiphp: Slot [20] registered [ 0.307113] acpiphp: Slot [21] registered [ 0.309076] acpiphp: Slot [22] registered [ 0.311122] acpiphp: Slot [23] registered [ 0.313098] acpiphp: Slot [24] registered [ 0.314096] acpiphp: Slot [25] registered [ 0.316080] acpiphp: Slot [26] registered [ 0.318084] acpiphp: Slot [27] registered [ 0.322181] acpiphp: Slot [28] registered [ 0.325091] acpiphp: Slot [29] registered [ 0.327109] acpiphp: Slot [30] registered [ 0.328106] acpiphp: Slot [31] registered [ 0.330093] PCI host bridge to bus 0000:00 [ 0.331015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.334022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349987] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353333] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362780] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378463] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387479] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.412017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417411] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.457015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.467959] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.495030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.517022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.530174] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.541022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.573023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.587903] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.599019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.608019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637051] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.652219] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.664022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.675022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.697017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711105] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.725027] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.734020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.761024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.778159] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782612] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.785560] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.791273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.798057] iommu: Default domain type: Passthrough [ 0.800500] SCSI subsystem initialized [ 0.802202] ACPI: bus type USB registered [ 0.804204] usbcore: registered new interface driver usbfs [ 0.806100] usbcore: registered new interface driver hub [ 0.809114] usbcore: registered new device driver usb [ 0.811299] pps_core: LinuxPPS API ver. 1 registered [ 0.814016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.818080] PTP clock support registered [ 0.820082] EDAC MC: Ver: 3.0.0 [ 0.822193] PCI: Using ACPI for IRQ routing [ 0.824610] NetLabel: Initializing [ 0.825014] NetLabel: domain hash size = 128 [ 0.826014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.827173] NetLabel: unlabeled traffic allowed by default [ 0.829485] vgaarb: loaded [ 0.831286] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.834013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.841000] clocksource: Switched to clocksource kvm-clock [ 0.949486] VFS: Disk quotas dquot_6.6.0 [ 0.951387] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954625] *** VALIDATE ramfs *** [ 0.956215] *** VALIDATE hugetlbfs *** [ 0.957578] pnp: PnP ACPI init [ 0.959688] pnp: PnP ACPI: found 6 devices [ 0.976788] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.980143] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.982547] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.984911] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.989596] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.992915] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.995989] NET: Registered protocol family 2 [ 0.998686] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.004650] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.008078] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.013661] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.017932] TCP: Hash tables configured (established 65536 bind 65536) [ 1.021128] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.024221] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027404] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030468] NET: Registered protocol family 1 [ 1.033489] RPC: Registered named UNIX socket transport module. [ 1.035396] RPC: Registered udp transport module. [ 1.037286] RPC: Registered tcp transport module. [ 1.038904] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041275] NET: Registered protocol family 44 [ 1.042787] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.044911] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.046865] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049451] PCI: CLS 0 bytes, default 64 [ 1.051157] Unpacking initramfs... [ 2.531333] debug: unmapping init [mem 0xffff987a7cc54000-0xffff987a7ffbffff] [ 2.535983] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.538665] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.541880] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.214305] Initialise system trusted keyrings [ 3.215936] Key type blacklist registered [ 3.219535] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.231523] zbud: loaded [ 3.235362] *** VALIDATE nfs *** [ 3.236102] *** VALIDATE nfs4 *** [ 3.246791] pstore: using deflate compression [ 3.259859] Platform Keyring initialized [ 3.392889] NET: Registered protocol family 38 [ 3.394783] Key type asymmetric registered [ 3.396327] Asymmetric key parser 'x509' registered [ 3.398036] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.400908] io scheduler mq-deadline registered [ 3.405038] io scheduler kyber registered [ 3.406671] io scheduler bfq registered [ 3.408490] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.411473] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.414326] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.417185] ACPI: Power Button [PWRF] [ 3.425177] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.432651] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.470251] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.494921] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.542273] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.574101] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.604645] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.612602] Non-volatile memory driver v1.3 [ 3.614413] Linux agpgart interface v0.103 [ 3.650387] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.655220] vda: detected capacity change from 0 to 74825728 [ 3.672218] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.675469] vdb: detected capacity change from 0 to 1073741824 [ 3.696891] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.699777] vdc: detected capacity change from 0 to 2621440000 [ 3.713443] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.716129] vdd: detected capacity change from 0 to 2621440000 [ 3.729105] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.731489] vde: detected capacity change from 0 to 4294967296 [ 3.745619] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.748149] vdf: detected capacity change from 0 to 4294967296 [ 3.755678] libphy: Fixed MDIO Bus: probed [ 3.761480] usbcore: registered new interface driver usbserial_generic [ 3.765429] usbserial: USB Serial support registered for generic [ 3.768472] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.774121] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.776267] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.779221] mousedev: PS/2 mouse device common for all mice [ 3.785636] rtc_cmos 00:05: RTC can wake from S4 [ 3.790107] rtc_cmos 00:05: registered as rtc0 [ 3.791722] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.795474] intel_pstate: CPU model not supported [ 3.798827] hid: raw HID events driver (C) Jiri Kosina [ 3.801520] usbcore: registered new interface driver usbhid [ 3.803856] usbhid: USB HID core driver [ 3.805973] drop_monitor: Initializing network drop monitor service [ 3.808358] Initializing XFRM netlink socket [ 3.808924] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.810483] NET: Registered protocol family 10 [ 3.811997] Segment Routing with IPv6 [ 3.820400] NET: Registered protocol family 17 [ 3.823896] mpls_gso: MPLS GSO support [ 3.825533] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.829172] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.829282] RAS: Correctable Errors collector initialized. [ 3.837431] AVX version of gcm_enc/dec engaged. [ 3.839288] AES CTR mode by8 optimization enabled [ 3.942541] sched_clock: Marking stable (3942478791, 0)->(4920149171, -977670380) [ 3.945284] registered taskstats version 1 [ 3.947419] Loading compiled-in X.509 certificates [ 3.948892] zswap: loaded using pool lzo/zbud [ 3.983194] Key type big_key registered [ 3.998881] Key type encrypted registered [ 4.000647] ima: No TPM chip found, activating TPM-bypass! [ 4.003453] ima: Allocated hash algorithm: sha1 [ 4.005531] ima: No architecture policies found [ 4.007654] evm: Initialising EVM extended attributes: [ 4.009861] evm: security.selinux [ 4.011435] evm: security.ima [ 4.012770] evm: security.capability [ 4.014555] evm: HMAC attrs: 0x1 [ 4.017569] rtc_cmos 00:05: setting system clock to 2026-09-01 14:03:58 UTC (1788271438) [ 4.026650] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 4.030435] debug: unmapping init [mem 0xffffffffa3582000-0xffffffffa3858fff] [ 4.041177] Write protecting the kernel read-only data: 28672k [ 4.044692] debug: unmapping init [mem 0xffffffffa1c03000-0xffffffffa1dfffff] [ 4.048434] debug: unmapping init [mem 0xffffffffa2514000-0xffffffffa25fffff] [ 4.094034] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.103725] systemd[1]: Detected virtualization kvm. [ 4.106220] systemd[1]: Detected architecture x86-64. [ 4.110153] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.141813] systemd[1]: No hostname configured. [ 4.143518] systemd[1]: Set hostname to . [ 4.145807] random: systemd: uninitialized urandom read (16 bytes read) [ 4.148301] systemd[1]: Initializing machine ID from random generator. [ 4.204824] random: ln: uninitialized urandom read (6 bytes read) [ 4.337768] random: systemd: uninitialized urandom read (16 bytes read) [ 4.341640] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.347878] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.360591] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.692458] device-mapper: uevent: version 1.0.3 [ 5.694660] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [[ 6.569217] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy [ 6.952154] virtio_net virtio0 ens2: renamed from eth0 Gatherer Daemon. [ 9.895739] scsi host0: ata_piix [ 9.952263] scsi host1: ata_piix [ 9.953803] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.957428] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.540331] random: crng init done [ 13.542110] random: 7 urandom warning(s) missed due to ratelimiting [ 15.925700] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.797440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.889119] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.283571] SELinux: Disabled at runtime. [ 20.375929] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.384961] systemd[1]: Detected virtualization kvm. [ 20.386918] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.192667] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.197379] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.211255] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.216210] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.224143] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.232271] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.242373] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel...[ 21.385401] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 22.501132] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.248345] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.341748] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.695886] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.003907] EDAC sbridge: Ver: 1.1.2 [ 26.613963] Key type dns_resolver registered [ 27.073361] NFS: Registering the id_resolver key type [ 27.075300] Key type id_resolver registered [ 27.076978] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-server login: [ 58.469373] libcfs: loading out-of-tree module taints kernel. [ 58.491327] Key type ._llcrypt registered [ 58.492607] Key type .llcrypt registered [ 58.540462] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 68.154062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 68.887176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.897899] alg: No test for adler32 (adler32-zlib) [ 69.967134] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 70.370717] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 71.999116] Key type lgssc registered [ 72.720405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.438224] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.465668] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 140.177504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.202733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.386502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.431602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.530860] Lustre: lustre-MDT0000: new disk, initializing [ 141.627240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.652953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.848216] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.481853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.561555] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.588985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.592702] Lustre: Skipped 1 previous similar message [ 158.712755] Lustre: lustre-MDT0001: new disk, initializing [ 158.794214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 158.846456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.865420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.359440] hrtimer: interrupt took 10429419 ns [ 163.565415] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.408782] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 177.108359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.546626] Lustre: lustre-OST0000: new disk, initializing [ 177.553634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.579666] Lustre: 8421:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.664200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.223856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 181.228900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 181.302460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 182.475475] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.090693] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.200893] Lustre: lustre-OST0001: new disk, initializing [ 194.206699] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.212170] Lustre: 9490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 194.266845] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 198.562049] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.340604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.346435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.408239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.966848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 212.499861] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 217.717479] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing check_logdir /tmp/testlogs/ [ 222.055512] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing yml_node [ 225.911377] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 228.007620] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 229.982513] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 231.438058] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 10:07:44 EDT 2026 [ 244.909859] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 246.191559] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 260.576487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 260.578204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 260.598239] Lustre: Skipped 3 previous similar messages [ 260.601695] Lustre: Skipped 2 previous similar messages [ 265.700219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 265.706741] Lustre: Skipped 4 previous similar messages [ 266.685809] Lustre: server umount lustre-MDT0000 complete [ 269.916568] LustreError: 10307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271704 with bad export cookie 16963198373133405838 [ 269.939990] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 269.941099] LustreError: 10307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 270.824727] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.829856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 270.842847] Lustre: Skipped 1 previous similar message [ 270.855509] Lustre: Skipped 1 previous similar message [ 274.847977] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 274.857237] Lustre: Skipped 1 previous similar message [ 276.280958] Lustre: server umount lustre-MDT0001 complete [ 279.479404] Lustre: server umount lustre-OST0000 complete [ 282.547700] Lustre: server umount lustre-OST0001 complete [ 288.542847] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 295.907337] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 329.300987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 329.521347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 329.570709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 329.674821] Lustre: lustre-MDT0000: new disk, initializing [ 329.785038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 329.816990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 333.529653] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 344.000328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.135833] Lustre: 15039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 344.271187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 344.286781] Lustre: Skipped 1 previous similar message [ 344.407485] Lustre: lustre-MDT0001: new disk, initializing [ 344.564830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.626735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 344.640959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 349.614523] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 360.133061] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 365.615337] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 372.410988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 372.637168] Lustre: lustre-OST0000: new disk, initializing [ 372.640725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 372.653276] Lustre: 17009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 372.747855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.693413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 374.702695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 374.860875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 380.635114] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 390.461167] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 395.141396] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 395.363479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 398.801294] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 398.994723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 405.479157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.479615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.484371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 405.486767] LustreError: Skipped 1 previous similar message [ 405.499084] Lustre: Skipped 1 previous similar message [ 406.597670] Lustre: server umount lustre-OST0000 complete [ 410.599254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 410.619335] Lustre: Skipped 1 previous similar message [ 415.714380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 415.736031] Lustre: Skipped 4 previous similar messages [ 416.485855] Lustre: server umount lustre-MDT0000 complete [ 420.246094] LustreError: 17008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271854 with bad export cookie 16963198373133407371 [ 420.250947] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.265834] LustreError: 17008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 420.643791] Lustre: server umount lustre-MDT0001 complete [ 424.205357] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:10:57 (1788271857) [ 426.647790] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 428.246802] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:11:01 (1788271861) [ 434.947147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.434349] LustreError: 19419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.453159] LustreError: 19419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 435.516632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.764322] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 440.801209] LustreError: 19420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.903990] LustreError: 19419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.111811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.453327] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 457.763604] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 462.758914] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 468.816881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.110706] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 481.394165] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 490.978894] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 490.988827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.018094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.644856] Lustre: server umount lustre-OST0000 complete [ 498.657207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 499.172557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.186916] Lustre: Skipped 2 previous similar messages [ 504.194700] Lustre: server umount lustre-MDT0000 complete [ 504.288903] LustreError: 19414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.322930] LustreError: 19414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 507.948176] LustreError: 19400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271942 with bad export cookie 16963198373133408253 [ 507.956596] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.961885] LustreError: 19400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 508.386620] Lustre: server umount lustre-MDT0001 complete [ 519.828838] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 522.650746] Key type lgssc unregistered [ 523.018206] LNet: 22952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.032642] LNetError: 22952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 523.047524] LNet: Removed LNI 192.168.203.154@tcp [ 524.126075] Key type .llcrypt unregistered [ 524.131697] Key type ._llcrypt unregistered [ 545.443044] Key type ._llcrypt registered [ 545.445993] Key type .llcrypt registered [ 545.566933] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 546.734243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 546.803340] alg: No test for adler32 (adler32-zlib) [ 547.915275] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 548.184566] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 549.903187] Key type lgssc registered [ 550.876021] Lustre: Echo OBD driver; http://www.lustre.org/ [ 557.888178] loop: module loaded [ 557.957738] loop0: detected capacity change from 0 to 204800000 [ 558.194529] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 558.201597] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 558.486149] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.626861] loop0: detected capacity change from 0 to 204800000 [ 563.780685] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.787932] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.795942] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 568.533847] loop0: detected capacity change from 0 to 204800000 [ 573.445679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 573.476886] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.759041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 574.810296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 574.872956] Lustre: lustre-MDT0000: new disk, initializing [ 574.929943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.948953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 579.728912] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 588.254263] loop1: detected capacity change from 0 to 204800000 [ 593.751972] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.006521] Lustre: lustre-OST07c6: new disk, initializing [ 594.017143] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 594.020795] Lustre: Skipped 1 previous similar message [ 594.029893] Lustre: 26139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 594.097179] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 600.317471] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 602.733373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 602.751899] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 602.792801] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 604.482366] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 604.494314] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 604.500892] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 604.505252] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 604.511933] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 604.519519] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 619.136931] Lustre: 25125:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 619.148254] Lustre: 25125:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 619.156808] Lustre: 25125:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 619.162905] Lustre: 25125:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 619.173654] Lustre: 25125:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 619.180723] Lustre: 25125:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 623.625500] Lustre: 25125:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 623.637109] Lustre: 25125:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 623.643928] Lustre: 25125:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 623.650883] Lustre: 25125:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 623.656767] Lustre: 25125:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 623.661181] Lustre: 25125:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 623.665543] Lustre: 25125:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 623.670751] Lustre: 25125:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 623.677502] Lustre: 25125:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 623.682102] Lustre: 25125:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 623.686519] Lustre: 25125:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 623.691136] Lustre: 25125:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.854705] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 626.860109] Lustre: 25124:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 626.864155] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 626.867644] Lustre: 25124:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.872033] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 626.877814] Lustre: 25124:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.882158] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 626.886790] Lustre: 25124:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.893232] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 626.899337] Lustre: 25124:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.905214] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 626.915656] Lustre: 25124:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 631.202458] Lustre: 26822:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 631.216827] Lustre: 26822:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 631.223838] Lustre: 26822:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 631.229765] Lustre: 26822:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 631.236071] Lustre: 26822:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 631.242624] Lustre: 26822:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 631.249466] Lustre: 26822:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 631.256970] Lustre: 26822:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 631.263504] Lustre: 26822:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 631.270092] Lustre: 26822:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 631.276291] Lustre: 26822:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 631.282023] Lustre: 26822:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 642.070092] Lustre: 26822:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 642.080323] Lustre: 26822:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 642.086178] Lustre: 26822:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 642.091507] Lustre: 26822:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 642.096662] Lustre: 26822:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 642.103159] Lustre: 26822:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 642.109428] Lustre: 26822:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 642.118386] Lustre: 26822:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 642.128874] Lustre: 26822:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 642.133942] Lustre: 26822:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 642.138925] Lustre: 26822:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 642.144636] Lustre: 26822:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 689.639942] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 689.645656] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 689.675733] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 694.755371] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 695.680383] Lustre: server umount lustre-OST07c6 complete [ 704.350408] Lustre: server umount lustre-MDT0000 complete [ 717.228756] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 719.437916] Key type lgssc unregistered [ 719.671534] LNet: 28699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.680425] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.707196] LNet: Removed LNI 192.168.203.154@tcp [ 720.413232] Key type .llcrypt unregistered [ 720.415303] Key type ._llcrypt unregistered [ 734.424461] Key type ._llcrypt registered [ 734.426230] Key type .llcrypt registered [ 734.486291] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 735.052180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.117071] alg: No test for adler32 (adler32-zlib) [ 736.116493] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 736.330409] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 737.983177] Key type lgssc registered [ 738.814410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 744.377394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 744.393168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.810235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.732111] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 756.195193] LustreError: 29948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.213408] LustreError: 29948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 756.283729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 756.600392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 759.478404] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 764.630874] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 768.338732] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 772.780960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 773.002378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.095477] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 781.568971] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 791.466946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 806.883407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 806.889279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.893930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 806.899403] LustreError: Skipped 1 previous similar message [ 806.920389] Lustre: Skipped 1 previous similar message [ 808.127842] Lustre: server umount lustre-OST0000 complete [ 812.001457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 812.003820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 812.010251] Lustre: Skipped 1 previous similar message [ 812.012737] Lustre: Skipped 1 previous similar message [ 817.123950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 817.130984] Lustre: Skipped 1 previous similar message [ 822.241340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 822.247980] Lustre: Skipped 1 previous similar message [ 825.311128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 825.425585] Lustre: server umount lustre-MDT0000 complete [ 827.360589] LustreError: 29948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.374907] LustreError: 29948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 827.863145] LustreError: 29931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272262 with bad export cookie 10625091832449532438 [ 827.865963] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 827.868885] LustreError: 29931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 828.075898] Lustre: server umount lustre-MDT0001 complete [ 835.780341] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 837.755756] Key type lgssc unregistered [ 837.989495] LNet: 33705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.000150] LNetError: 33705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.020787] LNet: Removed LNI 192.168.203.154@tcp [ 838.728604] Key type .llcrypt unregistered [ 838.733234] Key type ._llcrypt unregistered [ 850.838144] Key type ._llcrypt registered [ 850.839740] Key type .llcrypt registered [ 850.894827] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 851.483786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.508893] alg: No test for adler32 (adler32-zlib) [ 852.584477] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 852.841779] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 854.471224] Key type lgssc registered [ 855.364574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 864.928293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 864.983567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.630150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.960390] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 877.029612] LustreError: 34953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 877.051727] LustreError: 34953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 882.146425] LustreError: 34952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.268286] LustreError: 34953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 887.884256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.841272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.405062] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 909.074066] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 914.725983] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.251954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 921.716035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 929.468540] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 937.106516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 940.556241] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 957.507443] Lustre: server umount lustre-OST0000 complete [ 963.057259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.064202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 963.079842] Lustre: Skipped 1 previous similar message [ 963.096222] Lustre: Skipped 1 previous similar message [ 968.160601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.171959] Lustre: Skipped 1 previous similar message [ 972.345881] Lustre: server umount lustre-MDT0000 complete [ 973.281582] LustreError: 35681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 973.320090] LustreError: 35681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 974.303151] Lustre: 34267:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788272392/real 1788272392] req@000000000be6aa04 x1875139401646848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788272408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 974.344573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.151236] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272410 with bad export cookie 1466863155946468217 [ 976.156900] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.160979] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 976.608015] Lustre: server umount lustre-MDT0001 complete [ 990.262649] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 993.716499] Key type lgssc unregistered [ 994.198263] LNet: 38517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.219752] LNetError: 38517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.247721] LNet: Removed LNI 192.168.203.154@tcp [ 995.415135] Key type .llcrypt unregistered [ 995.417685] Key type ._llcrypt unregistered [ 1023.457138] Key type ._llcrypt registered [ 1023.459556] Key type .llcrypt registered [ 1023.552962] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1024.771735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.831350] alg: No test for adler32 (adler32-zlib) [ 1025.894592] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1026.178839] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1027.871507] Key type lgssc registered [ 1029.180416] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.403798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.436429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.874944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.176252] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1049.059188] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.090968] LustreError: 39936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1052.238689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.511602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.478412] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1065.441917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1071.430675] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1078.399584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1078.821807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.962797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1085.421182] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1093.681286] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.842091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1103.862735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.886807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.550552] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1107.431449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1107.446644] Lustre: Skipped 1 previous similar message [ 1110.133110] Lustre: server umount lustre-MDT0000 complete [ 1111.649450] LustreError: 41510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.365762] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272548 with bad export cookie 12538179112805006150 [ 1114.374978] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1114.384720] LustreError: 39915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1116.762500] LustreError: 39930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.780775] LustreError: 39930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1116.801096] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 1116.805340] Lustre: Skipped 3 previous similar messages [ 1117.672643] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.735674] LustreError: 39932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.738558] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1119.769162] LustreError: 39932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1119.783792] Lustre: Skipped 1 previous similar message [ 1120.846336] Lustre: server umount lustre-MDT0001 complete [ 1136.005747] Lustre: server umount lustre-OST0000 complete [ 1152.943263] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1156.309580] Key type lgssc unregistered [ 1156.572342] LNet: 43803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.581954] LNetError: 43803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.593424] LNet: Removed LNI 192.168.203.154@tcp [ 1157.560821] Key type .llcrypt unregistered [ 1157.562789] Key type ._llcrypt unregistered [ 1180.470610] Key type ._llcrypt registered [ 1180.474105] Key type .llcrypt registered [ 1180.543163] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1181.652772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.671519] alg: No test for adler32 (adler32-zlib) [ 1182.750177] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1183.053820] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1184.791331] Key type lgssc registered [ 1186.112908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.556892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.594357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.129594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.550733] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1205.746336] LustreError: 45219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1205.781710] LustreError: 45219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1209.695904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.280692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1214.853757] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1222.585270] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1228.490270] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.379600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1235.609502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.754498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1242.328785] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1249.416180] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1260.523486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1260.535351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.564720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.685613] Lustre: lustre-OST0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1261.698879] Lustre: Skipped 1 previous similar message [ 1264.974345] Lustre: server umount lustre-OST0000 complete [ 1286.113258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.115500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1286.128261] Lustre: Skipped 2 previous similar messages [ 1289.699979] Lustre: server umount lustre-MDT0000 complete [ 1291.240054] LustreError: 45219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.268306] LustreError: 45219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1293.940216] LustreError: 45201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272728 with bad export cookie 3761035475710391091 [ 1293.944529] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1293.952028] LustreError: 45201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1300.513441] Lustre: server umount lustre-MDT0001 complete [ 1311.756904] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1314.600532] Key type lgssc unregistered [ 1314.880670] LNet: 48931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.905149] LNetError: 48931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.936732] LNet: Removed LNI 192.168.203.154@tcp [ 1315.704909] Key type .llcrypt unregistered [ 1315.707184] Key type ._llcrypt unregistered [ 1337.725804] Key type ._llcrypt registered [ 1337.728134] Key type .llcrypt registered [ 1337.808162] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1338.612320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.741329] alg: No test for adler32 (adler32-zlib) [ 1339.876470] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1340.111221] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1341.775457] Key type lgssc registered [ 1342.930096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1350.318168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1350.336155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.815968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.943914] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1362.409033] LustreError: 50347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.431697] LustreError: 50347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1365.531724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.945838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.534768] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1378.587622] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1383.910944] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1389.561272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.834147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.680228] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1396.970522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1402.698476] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1408.939697] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1412.462617] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1424.857739] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0110h27m37s [ 1426.660595] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1431.147661] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1432.212649] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [ 1432.552962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1432.567153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1433.642456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.649715] Lustre: Skipped 5 previous similar messages [ 1434.972218] Lustre: server umount lustre-MDT0000 complete [ 1437.277109] LustreError: 50344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.595726] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1439.724681] LustreError: 53032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.738141] LustreError: 53032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1443.427773] LustreError: 51587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1445.513388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.775694] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.146625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1446.222201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1447.151888] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1451.113452] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1454.851675] Lustre: DEBUG MARKER: Wait for df (36738) ... [ 1456.662756] Lustre: DEBUG MARKER: done [ 1461.741056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.752525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.776769] Lustre: Skipped 2 previous similar messages [ 1461.791847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.759535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.763163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.118923] Lustre: server umount lustre-OST0000 complete [ 1472.993517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.996433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1473.011968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1473.011975] Lustre: Skipped 2 previous similar messages [ 1484.257740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.274314] Lustre: Skipped 6 previous similar messages [ 1485.279873] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.560185] Lustre: server umount lustre-MDT0000 complete [ 1489.314990] LustreError: 53032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.352924] LustreError: 53032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1489.527643] LustreError: 50328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272924 with bad export cookie 5872216960231343908 [ 1489.533608] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1489.542443] LustreError: 50328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1489.958810] Lustre: server umount lustre-MDT0001 complete [ 1499.977601] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1502.750618] Key type lgssc unregistered [ 1503.007725] LNet: 55273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.019443] LNetError: 55273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.034603] LNet: Removed LNI 192.168.203.154@tcp [ 1503.648948] Key type .llcrypt unregistered [ 1503.656863] Key type ._llcrypt unregistered [ 1540.787657] Key type ._llcrypt registered [ 1540.790213] Key type .llcrypt registered [ 1540.911611] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1542.268115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1542.416581] alg: No test for adler32 (adler32-zlib) [ 1543.793513] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1544.199664] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1545.959145] Key type lgssc registered [ 1547.482962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1556.132685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1556.149893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.564875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1557.626714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1557.877655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.372579] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1570.093228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1570.183910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1570.207995] Lustre: 57484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1570.543211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1575.665966] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1584.474360] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1591.333148] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1599.049746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.293156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1599.508687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.325051] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1609.907318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1615.497316] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1622.381352] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0110h30m55s [ 1623.744319] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1631.627502] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1634.386897] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1634.427763] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1654.930844] Lustre: lustre-MDT0000: Client 32d9b837-b2f6-4d76-8294-ec68d621944b (at 192.168.203.54@tcp) reconnecting [ 1674.527239] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1676.646451] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1679.287157] Lustre: DEBUG MARKER: done [ 1686.497520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1686.507464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.523393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.013573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.014525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.035282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.062475] Lustre: Skipped 1 previous similar message [ 1692.144214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.150591] Lustre: Skipped 1 previous similar message [ 1697.248960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.266499] Lustre: Skipped 1 previous similar message [ 1699.295222] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1699.397391] Lustre: server umount lustre-OST0000 complete [ 1703.905941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.922803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.934730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1709.592276] Lustre: server umount lustre-MDT0000 complete [ 1714.207335] LustreError: 59343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273148 with bad export cookie 4493616643979171744 [ 1714.223500] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.224816] LustreError: 59343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1714.475392] Lustre: server umount lustre-MDT0001 complete [ 1726.396939] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 1729.061574] Key type lgssc unregistered [ 1729.292380] LNet: 61817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.300171] LNetError: 61817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.348448] LNet: Removed LNI 192.168.203.154@tcp [ 1731.103977] Key type .llcrypt unregistered [ 1731.114618] Key type ._llcrypt unregistered [ 1776.649463] Key type ._llcrypt registered [ 1776.651811] Key type .llcrypt registered [ 1776.762128] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 1777.923566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.006342] alg: No test for adler32 (adler32-zlib) [ 1779.046383] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 1779.361371] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 1781.095152] Key type lgssc registered [ 1782.304688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1785.990113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.021473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1787.337958] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1787.373466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1787.564422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.544927] Lustre: Failing over lustre-MDT0000 [ 1788.830466] Lustre: server umount lustre-MDT0000 complete [ 1794.600056] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:33:48 (1788273228) [ 1801.331438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.768484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1806.312186] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1814.161789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.265084] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1814.321510] Lustre: 64626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1814.638632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.870985] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1827.133835] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1833.214859] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.875140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.111104] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1840.281966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1845.903246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1846.249022] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1853.032391] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1859.246950] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1865.698224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.703342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.719518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.724450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.728159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.739717] Lustre: Skipped 1 previous similar message [ 1866.755206] Lustre: Skipped 2 previous similar messages [ 1871.468402] Lustre: server umount lustre-MDT0000 complete [ 1875.335079] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273309 with bad export cookie 2050799276708493866 [ 1875.347617] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1875.351096] LustreError: 64617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1875.626954] Lustre: server umount lustre-MDT0001 complete [ 1879.258507] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1883.948341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1892.303119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1893.344131] Lustre: 63552:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273311/real 1788273311] req@00000000e0b5d7ca x1875140372634368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1893.386126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.119425] Lustre: 63553:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273315/real 1788273315] req@00000000e45203d3 x1875140372634496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273331 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1898.463234] Lustre: 63551:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273316/real 1788273316] req@00000000a1f4bd1c x1875140372634624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1899.557135] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1c75e71399d6830d [ 1899.568101] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 1899.917758] LustreError: 66474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.947485] LustreError: 66474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1900.013184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.105385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1900.961465] Lustre: 63551:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273319/real 1788273319] req@000000009e52b5fa x1875140372634880/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273335 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1903.009409] LustreError: 66475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.034226] LustreError: 66475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.059688] LustreError: 66475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1904.360269] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1906.145525] Lustre: 63552:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273324/real 1788273324] req@00000000e57dd523 x1875140372635008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1909.222834] LustreError: 66474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1909.243232] LustreError: 66474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1911.842362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.123765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.798227] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1917.426801] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1922.542710] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1924.231113] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1929.407944] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.890623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.894746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1937.897379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.915133] Lustre: Skipped 1 previous similar message [ 1942.225725] Lustre: server umount lustre-MDT0000 complete [ 1943.012283] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.030755] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1945.924082] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273380 with bad export cookie 2050799276708496141 [ 1945.924903] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.936848] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1946.298193] Lustre: server umount lustre-MDT0001 complete [ 1953.265714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1963.284952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.511210] Lustre: 63552:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788273382/real 1788273382] req@000000000e919995 x1875140372660736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788273398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1964.556908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.582489] Lustre: Skipped 1 previous similar message [ 1971.109197] LustreError: 66474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.196961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.280506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1975.670415] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1983.906168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.050299] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 1989.617202] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1994.353461] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1999.068596] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2008.039593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.040971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.051603] Lustre: Skipped 2 previous similar messages [ 2008.075970] Lustre: Skipped 4 previous similar messages [ 2011.798065] Lustre: server umount lustre-MDT0000 complete [ 2013.153205] LustreError: 72148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.185802] LustreError: 72148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2015.372897] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273449 with bad export cookie 2050799276708497520 [ 2015.376426] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.392040] LustreError: 64618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2015.717733] Lustre: server umount lustre-MDT0001 complete [ 2021.907135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2025.599799] Lustre: DEBUG MARKER: files compared the same [ 2037.990734] Lustre: server umount lustre-OST0000 complete [ 2053.090496] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2055.800317] Key type lgssc unregistered [ 2056.085836] LNet: 74357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.097693] LNetError: 74357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.117069] LNet: Removed LNI 192.168.203.154@tcp [ 2056.908635] Key type .llcrypt unregistered [ 2056.912777] Key type ._llcrypt unregistered [ 2079.072767] Key type ._llcrypt registered [ 2079.075421] Key type .llcrypt registered [ 2079.165789] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2080.047903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2080.210096] alg: No test for adler32 (adler32-zlib) [ 2081.295704] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2081.527597] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2083.191190] Key type lgssc registered [ 2084.141147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2090.788876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2090.804744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.188907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2095.881409] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2101.728763] LustreError: 75773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.741216] LustreError: 75773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2103.177072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.400456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2106.995739] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2113.192175] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2118.522497] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2124.610485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2124.879912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.211913] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug +malloc all [ 2136.434880] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2137.193646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2148.833783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2148.839898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.855175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2152.939841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.945094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2152.949021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.655658] Lustre: server umount lustre-OST0000 complete [ 2159.585456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2159.591884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.615357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.626312] Lustre: Skipped 1 previous similar message [ 2163.173601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.174210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.207482] Lustre: Skipped 1 previous similar message [ 2164.813556] Lustre: server umount lustre-MDT0000 complete [ 2168.287786] LustreError: 76502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.303323] LustreError: 76502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2168.837687] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273603 with bad export cookie 5993423699724101797 [ 2168.840288] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.849078] LustreError: 75755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2169.060635] Lustre: server umount lustre-MDT0001 complete [ 2178.804481] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2181.274353] Key type lgssc unregistered [ 2181.564607] LNet: 79308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.576286] LNetError: 79308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.598447] LNet: Removed LNI 192.168.203.154@tcp [ 2182.362977] Key type .llcrypt unregistered [ 2182.366064] Key type ._llcrypt unregistered [ 2202.131584] Key type ._llcrypt registered [ 2202.133613] Key type .llcrypt registered [ 2202.226985] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2203.014139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.115355] alg: No test for adler32 (adler32-zlib) [ 2204.174758] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2204.389653] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2206.040557] Key type lgssc registered [ 2206.981661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.229603] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.259820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.841231] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2233.951318] LustreError: 80678:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [ 2260.575580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.157049] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2272.181886] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2277.813481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2277.928964] Lustre: *** cfs_fail_loc=706, val=0*** [ 2278.175972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.233728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2280.923721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2283.496984] LustreError: 80703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.513441] LustreError: 80703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2287.473280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.775648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.306561] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2297.541147] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2302.505486] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2305.508087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.541753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2305.551586] Lustre: Skipped 1 previous similar message [ 2310.454119] Lustre: server umount lustre-OST0000 complete [ 2315.745868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.757292] Lustre: Skipped 1 previous similar message [ 2315.763153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2315.768991] Lustre: Skipped 1 previous similar message [ 2319.805462] Lustre: server umount lustre-MDT0000 complete [ 2320.865622] LustreError: 82865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2320.882530] LustreError: 82865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2322.437453] LustreError: 80684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273756 with bad export cookie 12081993951695787893 [ 2322.446257] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.453478] LustreError: 80684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2322.672786] Lustre: server umount lustre-MDT0001 complete [ 2330.740475] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2333.007889] Key type lgssc unregistered [ 2333.299626] LNet: 84298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.320450] LNetError: 84298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.375592] LNet: Removed LNI 192.168.203.154@tcp [ 2334.988184] Key type .llcrypt unregistered [ 2334.991210] Key type ._llcrypt unregistered [ 2353.897412] Key type ._llcrypt registered [ 2353.901043] Key type .llcrypt registered [ 2353.994583] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2354.778699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.799940] alg: No test for adler32 (adler32-zlib) [ 2355.811582] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2355.989124] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2357.631473] Key type lgssc registered [ 2358.514750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2364.358921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2364.375764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.125440] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2373.618776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.055126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.031249] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2379.233374] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.248503] LustreError: 86381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2382.307950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2382.518631] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.532278] LustreError: 86379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2382.533545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.585494] LustreError: 86385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.594926] LustreError: 86385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2386.451358] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2388.961422] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.979118] LustreError: 86384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2393.180659] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2393.190293] Lustre: MGS: Client 9cee8e56-9ac9-416a-bc71-516e93226d67 (at 0@lo) reconnecting [ 2393.210686] LustreError: 86395:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2393.430496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.501211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2397.098680] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2410.464263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.471229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.485285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2414.055985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2414.059843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2414.068202] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.431245] Lustre: server umount lustre-OST0000 complete [ 2419.169703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2419.169971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.185325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.189603] Lustre: Skipped 2 previous similar messages [ 2424.289962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2424.297667] Lustre: Skipped 1 previous similar message [ 2424.389510] Lustre: server umount lustre-MDT0000 complete [ 2427.326663] Lustre: server umount lustre-MDT0001 complete [ 2430.366479] Lustre: server umount MGS complete [ 2438.334234] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:44:31 (1788273871) [ 2458.410140] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 2464.746645] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2492.030841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.062503] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2501.938257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2502.163815] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2502.203298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2502.272556] Lustre: lustre-MDT0001: new disk, initializing [ 2502.325775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.349248] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2502.358450] Lustre: 93542:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2502.369582] LustreError: 93542:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2505.505132] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2513.001056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.168662] Lustre: lustre-OST0000: new disk, initializing [ 2513.173669] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2513.178585] Lustre: 94308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2513.250844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.411466] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2520.608819] Lustre: 94766:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b1f0b201 x1875140977661952/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788273971 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2523.895574] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2528.498752] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.507976] Lustre: MGS: Client e49e5960-895d-4eb0-a393-8eedd78bd5cc (at 0@lo) reconnecting [ 2528.545459] Lustre: 92810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2528.562702] LustreError: 93528:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2528.583830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2528.654575] Lustre: lustre-MDT0000: new disk, initializing [ 2528.773548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.796179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2529.695784] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2529.827866] Lustre: 95378:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003e686a68 x1875140977668096/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788273980 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2531.692572] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2538.984130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2538.993682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2539.026015] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2558.439329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.441597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2558.448267] Lustre: Skipped 2 previous similar messages [ 2564.562353] Lustre: server umount lustre-OST0000 complete [ 2568.677382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.680441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.690293] Lustre: Skipped 1 previous similar message [ 2568.696380] Lustre: Skipped 3 previous similar messages [ 2573.200468] Lustre: server umount lustre-MDT0000 complete [ 2573.792471] LustreError: 94315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.805927] LustreError: 94315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2581.742506] Lustre: server umount lustre-MDT0001 complete [ 2584.415488] Lustre: server umount MGS complete [ 2591.045563] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:47:04 (1788274024) [ 2601.666084] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2603.734465] Key type lgssc unregistered [ 2603.972872] LNet: 97879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.980334] LNetError: 97879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.995105] LNet: Removed LNI 192.168.203.154@tcp [ 2604.590855] Key type .llcrypt unregistered [ 2604.592958] Key type ._llcrypt unregistered [ 2615.708083] Key type ._llcrypt registered [ 2615.711833] Key type .llcrypt registered [ 2615.780183] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2616.430060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2616.451108] alg: No test for adler32 (adler32-zlib) [ 2617.423962] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2617.595559] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2619.223336] Key type lgssc registered [ 2620.101806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2624.974516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2624.987957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.286986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.092871] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2634.688448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.933169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2637.419407] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2642.051841] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2645.713151] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2650.394773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.644654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.700310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2654.709247] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2659.129399] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2664.931763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.932650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.937105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.942910] Lustre: Skipped 1 previous similar message [ 2666.349763] Lustre: server umount lustre-OST0000 complete [ 2668.797823] Lustre: server umount lustre-MDT0000 complete [ 2670.050211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.052343] LustreError: 99076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.058403] Lustre: Skipped 1 previous similar message [ 2670.073685] LustreError: 99076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2671.406303] LustreError: 99060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274105 with bad export cookie 4482178037280467605 [ 2671.410713] LustreError: 99060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2671.410981] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.626990] Lustre: server umount lustre-MDT0001 complete [ 2685.874354] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2690.875378] Key type lgssc unregistered [ 2691.170692] LNet: 102611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.174159] LNetError: 102611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.197732] LNet: Removed LNI 192.168.203.154@tcp [ 2692.675480] Key type .llcrypt unregistered [ 2692.678594] Key type ._llcrypt unregistered [ 2719.896919] Key type ._llcrypt registered [ 2719.906569] Key type .llcrypt registered [ 2720.106698] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2722.935032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2722.966204] alg: No test for adler32 (adler32-zlib) [ 2724.109414] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2724.490750] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2726.167171] Key type lgssc registered [ 2727.580476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2733.679783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2733.680553] LustreError: 103700:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2733.688453] Lustre: Skipped 1 previous similar message [ 2733.688481] LustreError: 103699:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2733.709342] LustreError: 103700:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2733.715868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.717136] LustreError: 103700:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2733.717158] LustreError: 103700:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2735.205818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.296028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.531693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2747.281110] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2752.296815] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lsmod [ 2755.187350] LustreError: 104900:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2755.220684] LustreError: 104907:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2755.230644] LustreError: 104900:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4962 [ 2755.239197] LustreError: 104900:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2755.249232] LustreError: 104900:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2755.258236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2755.700918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.286163] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.297738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.233324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.237381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.259335] Lustre: Skipped 1 previous similar message [ 2763.270302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.278499] Lustre: Skipped 1 previous similar message [ 2764.503599] Lustre: server umount lustre-MDT0000 complete [ 2766.817362] LustreError: 103737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.832848] LustreError: 103737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2767.509660] LustreError: 103717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274201 with bad export cookie 14904779987674952223 [ 2767.519147] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.519662] LustreError: 103717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2768.927296] LustreError: 104168:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001c0a7edb x1875141363519104/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2769.077965] Lustre: server umount lustre-MDT0001 complete [ 2781.925899] Lustre: server umount lustre-OST0000 complete [ 2789.809982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.194028] LustreError: 105971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.254806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.403845] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2795.518123] LustreError: 105972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.598718] LustreError: 105971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.594604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.992724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2807.293628] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2813.904291] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2819.198688] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2824.907245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.211489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.002735] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2836.563048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2837.537496] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2846.690371] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.700449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.703205] Lustre: Skipped 1 previous similar message [ 2846.716279] Lustre: Skipped 3 previous similar messages [ 2849.879206] Lustre: server umount lustre-OST0000 complete [ 2854.370808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.385117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.408904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.830374] Lustre: server umount lustre-MDT0000 complete [ 2862.052486] LustreError: 105971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.066721] LustreError: 105971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2862.650905] LustreError: 105951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274297 with bad export cookie 14904779987674952916 [ 2862.666883] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.670790] LustreError: 105951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2862.895265] Lustre: server umount lustre-MDT0001 complete [ 2872.533557] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 2874.820631] Key type lgssc unregistered [ 2875.048733] LNet: 109505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.055536] LNetError: 109505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.070528] LNet: Removed LNI 192.168.203.154@tcp [ 2875.889879] Key type .llcrypt unregistered [ 2875.891759] Key type ._llcrypt unregistered [ 2895.211589] Key type ._llcrypt registered [ 2895.213083] Key type .llcrypt registered [ 2895.300504] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 2896.062749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2896.135250] alg: No test for adler32 (adler32-zlib) [ 2897.118319] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 2897.286279] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 2898.927163] Key type lgssc registered [ 2899.886894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2907.175967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2907.206326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.657463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.206655] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2918.885283] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.920596] LustreError: 110902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2921.594891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.113818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.256311] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2933.916831] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2939.945515] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2947.021672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.460084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.602344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2953.946561] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 2960.784777] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2970.150565] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2973.938091] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2979.297252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2979.309854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.331607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.370757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2982.372291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2982.390888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2982.408513] Lustre: Skipped 1 previous similar message [ 2984.420390] Lustre: server umount lustre-OST0000 complete [ 2988.512976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2988.527571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.553994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.611266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.622066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.643538] Lustre: Skipped 1 previous similar message [ 2994.564395] Lustre: server umount lustre-MDT0000 complete [ 2997.734235] LustreError: 110901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.771425] LustreError: 110901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2998.482546] LustreError: 111638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274432 with bad export cookie 13303132666650120251 [ 2998.486094] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2998.497064] LustreError: 111638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2998.781628] Lustre: server umount lustre-MDT0001 complete [ 3011.859170] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3015.519845] Key type lgssc unregistered [ 3016.033754] LNet: 114547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.046807] LNetError: 114547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3017.061726] LNet: Removed LNI 192.168.203.154@tcp [ 3018.182036] Key type .llcrypt unregistered [ 3018.183970] Key type ._llcrypt unregistered [ 3035.118247] Key type ._llcrypt registered [ 3035.120388] Key type .llcrypt registered [ 3035.224706] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3036.418149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.477097] alg: No test for adler32 (adler32-zlib) [ 3037.521182] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 3037.756268] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3039.392074] Key type lgssc registered [ 3040.391813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3049.479768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3049.500705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.461734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3058.004560] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3060.714701] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.749977] LustreError: 115772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3065.827746] LustreError: 115771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.858357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.480449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3073.324448] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3080.573758] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.106349] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3093.522340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.823226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.879372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3099.252780] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3108.339701] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3121.635621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3121.645978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3121.660205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.681081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3123.684627] LustreError: Skipped 1 previous similar message [ 3123.686320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.691078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3123.705497] Lustre: Skipped 1 previous similar message [ 3127.832071] Lustre: server umount lustre-OST0000 complete [ 3133.955045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3133.962666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3133.979371] Lustre: Skipped 1 previous similar message [ 3138.908885] Lustre: server umount lustre-MDT0000 complete [ 3139.046087] LustreError: 115767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.083135] LustreError: 115767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3142.922698] LustreError: 118775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274577 with bad export cookie 114838378615244639 [ 3142.938214] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3143.197505] Lustre: server umount lustre-MDT0001 complete [ 3152.729800] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3155.786939] Key type lgssc unregistered [ 3156.140043] LNet: 119328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3156.149856] LNetError: 119328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3156.189668] LNet: Removed LNI 192.168.203.154@tcp [ 3157.801563] Key type .llcrypt unregistered [ 3157.806075] Key type ._llcrypt unregistered [ 3184.032558] Key type ._llcrypt registered [ 3184.037615] Key type .llcrypt registered [ 3184.133897] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3185.223547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3185.561075] alg: No test for adler32 (adler32-zlib) [ 3186.786730] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 3187.019716] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3188.665249] Key type lgssc registered [ 3190.233687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3200.213844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3200.239871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3201.608344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3205.384737] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3211.748763] LustreError: 120746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.765655] LustreError: 120746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3213.146048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.440549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3217.270600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3223.326337] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3228.175290] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3233.875786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3234.254771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.588690] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3246.573637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3246.995732] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3253.737778] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3257.215919] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3257.227604] Lustre: Skipped 2 previous similar messages [ 3261.336417] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3261.341467] Lustre: Skipped 2 previous similar messages [ 3263.565464] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3263.586435] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3271.758256] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3271.767123] Lustre: Skipped 2 previous similar messages [ 3279.954154] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.54@tcp 192.168.0.[2,10]@tcp [ 3284.610149] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.54/24@tcp in log lustre-MDT0000 [ 3284.617776] Lustre: Skipped 2 previous similar messages [ 3294.216477] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3294.219461] Lustre: Skipped 1 previous similar message [ 3302.131199] Lustre: server umount lustre-OST0000 complete [ 3308.000976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.003854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.010573] Lustre: Skipped 1 previous similar message [ 3308.018945] Lustre: Skipped 1 previous similar message [ 3313.129796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.134261] Lustre: Skipped 1 previous similar message [ 3315.468346] Lustre: server umount lustre-MDT0000 complete [ 3318.246714] LustreError: 120745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.286047] LustreError: 120745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3318.980284] LustreError: 120726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274753 with bad export cookie 13598467590692210545 [ 3319.000248] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3319.001660] LustreError: 120726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3329.626493] Lustre: server umount lustre-MDT0001 complete [ 3341.590360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3346.019584] Key type lgssc unregistered [ 3346.491603] LNet: 125781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3346.500705] LNetError: 125781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3346.522546] LNet: Removed LNI 192.168.203.154@tcp [ 3347.998270] Key type .llcrypt unregistered [ 3348.004937] Key type ._llcrypt unregistered [ 3383.560962] Key type ._llcrypt registered [ 3383.564217] Key type .llcrypt registered [ 3383.683431] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3385.298504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3385.321262] alg: No test for adler32 (adler32-zlib) [ 3386.386484] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 3386.645821] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3388.319325] Key type lgssc registered [ 3389.584692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3397.048637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3397.069657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.575179] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3398.598674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.602126] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.54/24@tcp [ 3403.332650] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3408.865941] LustreError: 127345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.878695] LustreError: 127345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3411.535200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.782053] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3411.809151] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.54/24@tcp [ 3411.850365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3415.915330] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3423.443769] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3429.185439] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3435.776708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3436.204378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.895472] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3444.397792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3448.551894] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3458.019461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3458.032385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.033651] LustreError: Skipped 1 previous similar message [ 3458.037895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3458.061909] Lustre: Skipped 1 previous similar message [ 3460.065762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.070134] Lustre: Skipped 2 previous similar messages [ 3463.973818] Lustre: server umount lustre-OST0000 complete [ 3470.314469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.329031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.334267] Lustre: Skipped 1 previous similar message [ 3474.777945] Lustre: server umount lustre-MDT0000 complete [ 3475.433590] LustreError: 127344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3475.458874] LustreError: 127344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3479.461820] LustreError: 127327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274913 with bad export cookie 16719871566513486717 [ 3479.478812] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3479.486796] LustreError: 127327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3479.940762] Lustre: server umount lustre-MDT0001 complete [ 3491.237129] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3493.696108] Key type lgssc unregistered [ 3493.962867] LNet: 130949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3493.975822] LNetError: 130949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3494.004801] LNet: Removed LNI 192.168.203.154@tcp [ 3494.809430] Key type .llcrypt unregistered [ 3494.814896] Key type ._llcrypt unregistered [ 3518.706950] Key type ._llcrypt registered [ 3518.707941] Key type .llcrypt registered [ 3518.832424] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3538.386984] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3539.472209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3539.509175] alg: No test for adler32 (adler32-zlib) [ 3540.874438] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 3541.313993] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3543.065610] Key type lgssc registered [ 3544.387121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3581.064780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3581.106758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3582.379991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3582.424451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3582.534866] Lustre: lustre-MDT0000: new disk, initializing [ 3582.633626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3582.659427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3588.592723] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3602.018123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3602.153924] Lustre: 134348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3602.246614] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3602.252085] Lustre: Skipped 1 previous similar message [ 3602.337033] Lustre: lustre-MDT0001: new disk, initializing [ 3602.479122] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.514829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3602.519628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3606.835751] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3615.664768] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3621.023211] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3628.201337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.417886] Lustre: lustre-OST0000: new disk, initializing [ 3628.421758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3628.430062] Lustre: 136315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3628.536764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.378346] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3637.888045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3637.898046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3638.028124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3646.668063] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.235390] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3651.531332] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3655.841576] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3656.036403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3658.725600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.727063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.729735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.751449] LustreError: Skipped 1 previous similar message [ 3658.786336] Lustre: Skipped 1 previous similar message [ 3663.858483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3663.887522] Lustre: Skipped 2 previous similar messages [ 3664.269417] Lustre: server umount lustre-OST0000 complete [ 3668.963846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3668.966709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.981107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.983205] Lustre: Skipped 1 previous similar message [ 3668.990220] Lustre: Skipped 1 previous similar message [ 3674.081341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3674.088124] Lustre: Skipped 1 previous similar message [ 3675.146265] Lustre: server umount lustre-MDT0000 complete [ 3679.782445] LustreError: 136314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275114 with bad export cookie 13639773196898762033 [ 3679.797371] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.805875] LustreError: 136314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3680.291818] Lustre: server umount lustre-MDT0001 complete [ 3688.451821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3688.887613] LustreError: 138535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.900983] LustreError: 138535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3688.961405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.438594] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3694.053306] LustreError: 138536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3699.170710] LustreError: 138535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3701.898535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.219754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3706.780945] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3713.890601] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3719.083754] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3726.438842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3726.928334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3735.070129] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3744.379622] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3749.614880] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3749.787304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3753.968052] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3754.211317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3762.487601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3762.693409] Lustre: lustre-OST0001: new disk, initializing [ 3762.698381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3762.705167] Lustre: 141906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3762.801930] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3768.629854] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3772.495114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3772.512075] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3772.676875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3776.647942] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3776.865630] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3780.821279] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3781.026128] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3785.627549] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a66c8dd8000.ost_server_uuid 50 [ 3787.111845] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a66c8dd8000.ost_server_uuid in IDLE state after 0 sec [ 3796.224943] Lustre: server umount lustre-OST0001 complete [ 3798.498082] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.505332] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.527283] LustreError: 140421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3798.567097] LustreError: 140421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3800.033436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3801.058437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3801.071074] LustreError: Skipped 1 previous similar message [ 3801.074318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3801.093376] Lustre: Skipped 2 previous similar messages [ 3803.618980] LustreError: 140356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3803.646162] LustreError: 140356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3805.737847] Lustre: server umount lustre-OST0000 complete [ 3813.860322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.860980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3813.884399] Lustre: Skipped 1 previous similar message [ 3813.901019] Lustre: Skipped 4 previous similar messages [ 3824.095605] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3824.384839] Lustre: server umount lustre-MDT0000 complete [ 3828.453836] LustreError: 138515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275262 with bad export cookie 13639773196898762922 [ 3828.457126] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3828.461852] LustreError: 138515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3828.794053] Lustre: server umount lustre-MDT0001 complete [ 3851.851839] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 3855.582863] Key type lgssc unregistered [ 3855.958782] LNet: 144912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3855.974434] LNetError: 144912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3855.999938] LNet: Removed LNI 192.168.203.154@tcp [ 3857.138340] Key type .llcrypt unregistered [ 3857.145093] Key type ._llcrypt unregistered [ 3894.970712] Key type ._llcrypt registered [ 3894.972950] Key type .llcrypt registered [ 3895.039431] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 3907.070712] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 3907.669553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3907.732066] alg: No test for adler32 (adler32-zlib) [ 3908.795877] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 3909.014040] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 3910.712677] Key type lgssc registered [ 3911.927482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3947.880332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3947.898979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3949.181343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3949.250661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3949.328985] Lustre: lustre-MDT0000: new disk, initializing [ 3949.437170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3949.454342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3953.832095] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3964.683238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.850992] Lustre: 149047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.946378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3964.955189] Lustre: Skipped 1 previous similar message [ 3964.978752] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3965.133768] Lustre: lustre-MDT0001: new disk, initializing [ 3965.260223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.327907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3965.345086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3969.739691] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 3982.159587] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3989.216287] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3996.426374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3996.726226] Lustre: lustre-OST0000: new disk, initializing [ 3996.729987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3996.734023] Lustre: 151014:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3996.847799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.124682] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4005.498321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4005.522228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4005.639820] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4013.458240] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4018.861288] Lustre: Failing over lustre-OST0000 [ 4019.084247] Lustre: server umount lustre-OST0000 complete [ 4036.576025] Lustre: 147272:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788275455/real 1788275455] req@000000002d9624ca x1875142606096384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788275471 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4036.586887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.605024] Lustre: 147272:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4037.992430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4038.111090] Lustre: 147270:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788275455/real 1788275455] req@00000000fe43d385 x1875142606096128/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788275471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4038.146794] Lustre: 147270:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4038.352814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4038.375632] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4042.394243] Lustre: 147270:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788275460/real 1788275460] req@00000000084a2d86 x1875142606097536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788275476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4042.458617] Lustre: 147270:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4044.741912] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4047.463263] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4047.996842] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4048.003079] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4048.376544] Lustre: Failing over lustre-MDT0000 [ 4048.686861] Lustre: server umount lustre-MDT0000 complete [ 4049.375145] Lustre: 147271:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788275465/real 1788275465] req@00000000c370e4dc x1875142606098688/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788275481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4049.406151] Lustre: 147271:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4052.448289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4052.460849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.477368] Lustre: Skipped 1 previous similar message [ 4053.983256] Lustre: 147273:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788275471/real 1788275471] req@000000000a002f36 x1875142606099840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788275487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4058.094614] LustreError: 149053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.114855] LustreError: 149053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4063.203110] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.218325] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4068.328018] LustreError: 150137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.353991] LustreError: 150137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4068.384548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4068.645039] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.182810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4069.285305] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4073.557581] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4074.527575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4074.550992] Lustre: Skipped 1 previous similar message [ 4074.612711] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4074.718498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4075.913614] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4089.318681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4089.334631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.350169] Lustre: Skipped 2 previous similar messages [ 4089.358661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4093.783505] Lustre: server umount lustre-OST0000 complete [ 4100.067545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.074758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.086480] Lustre: Skipped 2 previous similar messages [ 4100.090780] Lustre: Skipped 2 previous similar messages [ 4105.184707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.192793] Lustre: Skipped 1 previous similar message [ 4110.304754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4110.308553] Lustre: Skipped 1 previous similar message [ 4113.375954] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4115.425078] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4115.451447] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4115.581119] Lustre: server umount lustre-MDT0000 complete [ 4118.701043] LustreError: 152274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275553 with bad export cookie 15977480282843956822 [ 4118.713970] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4118.727100] LustreError: 152274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4119.057472] Lustre: server umount lustre-MDT0001 complete [ 4130.124053] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 4132.718539] Key type lgssc unregistered [ 4132.984970] LNet: 154681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4133.000849] LNetError: 154681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4133.027053] LNet: Removed LNI 192.168.203.154@tcp [ 4133.903514] Key type .llcrypt unregistered [ 4133.907658] Key type ._llcrypt unregistered [ 4171.417262] Key type ._llcrypt registered [ 4171.422357] Key type .llcrypt registered [ 4171.562711] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4172.528717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4172.576069] alg: No test for adler32 (adler32-zlib) [ 4173.523873] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 4173.714540] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 4175.423166] Key type lgssc registered [ 4176.877949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4184.792994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4184.812512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4186.196956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4191.773144] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4196.320397] LustreError: 156215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.333271] LustreError: 156215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4201.449497] LustreError: 156216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.157335] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4205.934437] LustreError: 156244:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4205.944101] LustreError: 156244:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4205.960297] LustreError: 156244:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4206.287914] Lustre: server umount lustre-MDT0000 complete [ 4223.591590] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4235.487786] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4281.828515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4282.248329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4282.298926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4282.528542] Lustre: lustre-MDT0000: new disk, initializing [ 4282.750238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4282.795709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4288.292343] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4299.622403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.783491] Lustre: 159549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4299.863318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4299.871477] Lustre: Skipped 1 previous similar message [ 4299.967476] Lustre: lustre-MDT0001: new disk, initializing [ 4300.070775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4300.101903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4300.111826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4305.500850] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4316.603170] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4323.222278] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4330.381173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4330.637346] Lustre: lustre-OST0000: new disk, initializing [ 4330.643373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4330.651626] Lustre: 161520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4330.720825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4337.682173] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4339.283975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4339.297340] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4339.507377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4347.308746] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4350.992970] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4352.851983] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:16:26 (1788275786) [ 4631.525369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.527084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.542401] Lustre: Skipped 1 previous similar message [ 4631.561806] Lustre: Skipped 2 previous similar messages [ 4636.640724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4636.650144] Lustre: Skipped 2 previous similar messages [ 4641.759951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4641.766395] Lustre: Skipped 2 previous similar messages [ 4644.319290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4644.520256] Lustre: server umount lustre-MDT0000 complete [ 4646.880331] LustreError: 162241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.890175] LustreError: 162241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4648.643165] LustreError: 159543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276083 with bad export cookie 8702904737539925192 [ 4648.645225] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.661182] LustreError: 159543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4652.000811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.002821] LustreError: 160006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.020612] Lustre: Skipped 1 previous similar message [ 4652.046906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4655.127872] Lustre: server umount lustre-MDT0001 complete [ 4665.459849] Lustre: server umount lustre-OST0000 complete [ 4676.416662] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4686.511297] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4731.145814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.434560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4731.472881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4731.593183] Lustre: lustre-MDT0000: new disk, initializing [ 4731.677668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.700262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4736.812712] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4747.279916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.401598] Lustre: 166199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4747.471698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4747.474932] Lustre: Skipped 1 previous similar message [ 4747.581554] Lustre: lustre-MDT0001: new disk, initializing [ 4747.693926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4747.750054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4747.764477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.545937] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4762.190788] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4768.152765] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4774.950718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.254361] Lustre: lustre-OST0000: new disk, initializing [ 4775.258801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4775.265773] Lustre: 168168:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4775.361217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4782.726924] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4783.183550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4783.195745] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4783.337797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4792.687299] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4797.335319] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4797.563434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4802.280848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4802.485855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4809.195924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4809.196822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4809.209329] LustreError: Skipped 1 previous similar message [ 4809.211526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4809.222617] Lustre: Skipped 1 previous similar message [ 4809.252325] Lustre: Skipped 2 previous similar messages [ 4810.650731] Lustre: server umount lustre-OST0000 complete [ 4814.303872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4814.305351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.312776] Lustre: Skipped 1 previous similar message [ 4819.428086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.433815] Lustre: Skipped 3 previous similar messages [ 4820.566714] Lustre: server umount lustre-MDT0000 complete [ 4824.298360] LustreError: 166190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276258 with bad export cookie 8702904737540084022 [ 4824.301063] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.308788] LustreError: 166190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4824.983275] Lustre: server umount lustre-MDT0001 complete [ 4834.827693] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:24:27 (1788276267) [ 4850.949577] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 4860.490504] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 4895.646247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4895.936258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4895.970602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4896.057616] Lustre: lustre-MDT0000: new disk, initializing [ 4896.153343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.177176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4900.291735] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4910.507222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.662950] Lustre: 172924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4911.015849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4915.725298] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4925.440384] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4931.789452] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4939.104169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4939.591907] Lustre: lustre-OST0000: new disk, initializing [ 4939.596450] Lustre: Skipped 1 previous similar message [ 4939.602577] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4939.608641] Lustre: Skipped 2 previous similar messages [ 4939.615262] Lustre: 174909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4944.974783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4944.987403] Lustre: Skipped 1 previous similar message [ 4945.024425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4945.156787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4945.564884] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 4956.270887] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4965.864241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4965.865717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4965.896444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4965.900955] Lustre: Skipped 1 previous similar message [ 4971.817303] Lustre: server umount lustre-OST0000 complete [ 4977.633075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4977.649350] LustreError: Skipped 1 previous similar message [ 4977.659798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.674720] Lustre: Skipped 1 previous similar message [ 4983.235064] Lustre: server umount lustre-MDT0000 complete [ 4986.337357] LustreError: 172929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.348314] LustreError: 172929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4986.897214] LustreError: 172916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276421 with bad export cookie 8702904737540084904 [ 4986.899797] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.913027] LustreError: 172916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4987.295277] Lustre: server umount lustre-MDT0001 complete [ 4995.493191] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:27:08 (1788276428) [ 5010.332228] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5018.723798] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5055.204845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.466549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5055.489579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5055.610832] Lustre: lustre-MDT0000: new disk, initializing [ 5055.746993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5055.757089] Lustre: Skipped 2 previous similar messages [ 5055.785341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5060.144474] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5071.003938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.160975] Lustre: 179235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.488104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5076.411743] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5087.537749] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5093.760137] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5100.894863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.250499] Lustre: 181214:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5108.625473] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5109.880618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5110.080448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5117.943201] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5125.602989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5125.607982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.619403] LustreError: Skipped 1 previous similar message [ 5125.624485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5125.636719] Lustre: Skipped 2 previous similar messages [ 5125.643131] Lustre: Skipped 5 previous similar messages [ 5131.717083] Lustre: server umount lustre-OST0000 complete [ 5137.897996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5142.674354] Lustre: server umount lustre-MDT0000 complete [ 5146.081973] LustreError: 180085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.102891] LustreError: 180085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5146.675804] LustreError: 179228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276581 with bad export cookie 8702904737540085989 [ 5146.679801] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.690972] LustreError: 179228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5147.112651] Lustre: server umount lustre-MDT0001 complete [ 5157.941247] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5161.075879] Key type lgssc unregistered [ 5161.318618] LNet: 183074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5161.348198] LNetError: 183074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5161.379738] LNet: Removed LNI 192.168.203.154@tcp [ 5162.265741] Key type .llcrypt unregistered [ 5162.268423] Key type ._llcrypt unregistered [ 5184.159222] Key type ._llcrypt registered [ 5184.161987] Key type .llcrypt registered [ 5184.271623] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5185.388692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5185.498305] alg: No test for adler32 (adler32-zlib) [ 5186.504491] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 5186.723185] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5188.431636] Key type lgssc registered [ 5189.740076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5197.086405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5197.110660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.639515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5203.106971] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5209.059033] LustreError: 184490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.090844] LustreError: 184490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5212.073712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5212.388849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5217.141516] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5227.051457] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5234.967480] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5242.639042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.990071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5244.045366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5249.869874] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5256.438846] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.108735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.120137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.135687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5278.179554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5278.181650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5278.184938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.319910] Lustre: server umount lustre-OST0000 complete [ 5288.416817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5288.442281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5288.448390] Lustre: Skipped 2 previous similar messages [ 5290.821763] Lustre: server umount lustre-MDT0000 complete [ 5293.543604] LustreError: 184485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.568256] LustreError: 184485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5294.221291] LustreError: 184470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276728 with bad export cookie 10330763477960247154 [ 5294.224487] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5294.228955] LustreError: 184470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5294.516343] Lustre: server umount lustre-MDT0001 complete [ 5304.797618] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 5307.898896] Key type lgssc unregistered [ 5308.179954] LNet: 188047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5308.188412] LNetError: 188047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5309.222856] LNet: Removed LNI 192.168.203.154@tcp [ 5310.874242] Key type .llcrypt unregistered [ 5310.877426] Key type ._llcrypt unregistered [ 5337.802511] Key type ._llcrypt registered [ 5337.806727] Key type .llcrypt registered [ 5337.956395] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5339.073621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5339.186358] alg: No test for adler32 (adler32-zlib) [ 5340.312749] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 5340.644046] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 5342.383759] Key type lgssc registered [ 5343.623587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5351.990880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5352.023861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.783816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.805132] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5364.198575] LustreError: 189466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.224539] LustreError: 189466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5368.475613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.000484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5373.399087] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5381.209355] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5387.034932] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5394.434176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5394.802864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.902927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5401.297973] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5408.569569] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5419.497206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.498631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5419.511812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5419.518186] Lustre: Skipped 1 previous similar message [ 5420.755789] Lustre: server umount lustre-OST0000 complete [ 5428.735478] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9a66c309e800.ost_server_uuid 50 [ 5440.632830] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5455.331668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5455.345860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5455.351291] Lustre: Skipped 2 previous similar messages [ 5460.449336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5460.458036] Lustre: Skipped 1 previous similar message [ 5461.054569] Lustre: server umount lustre-MDT0000 complete [ 5464.725500] LustreError: 189448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276899 with bad export cookie 16983664832004387149 [ 5464.728658] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.737517] LustreError: 189448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5465.057541] Lustre: server umount lustre-MDT0001 complete [ 5473.971609] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:35:07 (1788276907) [ 5480.981052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5481.523533] LustreError: 193406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5481.657515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.384528] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5487.074832] LustreError: 193407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5492.195065] LustreError: 193406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5496.165380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.635761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5501.404434] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5509.977851] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5515.468539] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5522.710775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5523.073651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.163073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5529.581866] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5536.499401] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5543.684720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.974807] Lustre: lustre-OST0001: new disk, initializing [ 5543.979994] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5543.990480] Lustre: 196248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5544.122476] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5550.457498] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5551.169733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5551.183052] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5551.340261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5559.112211] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5563.366138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5563.367913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5563.372701] LustreError: Skipped 2 previous similar messages [ 5563.382106] Lustre: Skipped 1 previous similar message [ 5563.382889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5563.392671] Lustre: Skipped 1 previous similar message [ 5568.548190] Lustre: server umount lustre-OST0000 complete [ 5570.146657] LustreError: 196642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.139428] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5577.465699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5580.388634] LustreError: 195226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.440736] LustreError: 195226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5584.004948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5584.373734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5597.681100] LustreError: 195228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5597.700257] LustreError: 195228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5602.790867] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5602.795507] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.824224] Lustre: Skipped 1 previous similar message [ 5602.836966] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5602.850896] Lustre: Skipped 4 previous similar messages [ 5604.359892] Lustre: server umount lustre-OST0001 complete [ 5609.443157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5609.450710] LustreError: Skipped 1 previous similar message [ 5609.452925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.468403] Lustre: Skipped 1 previous similar message [ 5613.026476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.034697] Lustre: Skipped 2 previous similar messages [ 5614.716301] Lustre: server umount lustre-MDT0000 complete [ 5618.302692] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277052 with bad export cookie 16983664832004388213 [ 5618.307349] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.312612] LustreError: 193387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5618.586908] Lustre: server umount lustre-MDT0001 complete [ 5643.179691] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:37:56 (1788277076) [ 5649.947256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5650.276592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5650.349128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5650.624760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5654.706281] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5662.663169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.779786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5662.812793] Lustre: 199684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5667.323191] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5675.017137] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.586127] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5686.904309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.127434] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5693.013140] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5697.132251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5699.729499] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5706.097543] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5706.168521] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5712.516516] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5721.489456] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5727.217456] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.229828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5727.238597] Lustre: Skipped 2 previous similar messages [ 5727.259050] Lustre: Skipped 1 previous similar message [ 5730.970392] Lustre: server umount lustre-OST0000 complete [ 5732.336869] LustreError: 201534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.372946] LustreError: 201534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5747.692457] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.694076] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5753.678341] Lustre: server umount lustre-OST0001 complete [ 5760.485696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5760.495546] LustreError: Skipped 1 previous similar message [ 5760.505239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.508122] Lustre: Skipped 5 previous similar messages [ 5764.463038] Lustre: server umount lustre-MDT0000 complete [ 5768.140945] LustreError: 199677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277202 with bad export cookie 16983664832004389627 [ 5768.144603] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5768.150068] LustreError: 199677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5768.654971] Lustre: server umount lustre-MDT0001 complete [ 5793.126424] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:40:26 (1788277226) [ 5811.409770] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 5822.147656] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 5861.729989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.955737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5861.986734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5862.121329] Lustre: lustre-MDT0000: new disk, initializing [ 5862.179578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5862.184545] Lustre: Skipped 3 previous similar messages [ 5862.217651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5867.384206] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5878.048623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.204397] Lustre: 207790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5878.257769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5878.262446] Lustre: Skipped 1 previous similar message [ 5878.382852] Lustre: lustre-MDT0001: new disk, initializing [ 5878.596046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5878.624603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5882.789465] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5891.817168] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5898.360854] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5906.124207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.329477] Lustre: lustre-OST0000: new disk, initializing [ 5906.335610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5906.347937] Lustre: 209757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5912.140788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5912.159553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5912.251406] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5912.316569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5920.848660] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5925.217548] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5925.412969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5929.999580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5930.201327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5933.024476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.029388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.041166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.041173] Lustre: Skipped 2 previous similar messages [ 5933.069715] Lustre: Skipped 4 previous similar messages [ 5946.335210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5946.556474] Lustre: server umount lustre-OST0000 complete [ 5956.573158] Lustre: server umount lustre-MDT0000 complete [ 5958.624936] LustreError: 207800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.648374] LustreError: 207800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5960.450863] LustreError: 207780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277394 with bad export cookie 16983664832004391188 [ 5960.452098] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5960.458131] LustreError: 207780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5960.790546] Lustre: server umount lustre-MDT0001 complete [ 5969.597787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5974.785873] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5983.979439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.045570] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 5997.579358] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6003.931769] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6011.326370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6011.749187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.756418] Lustre: Skipped 4 previous similar messages [ 6018.483571] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6026.550981] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6032.017933] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6032.237962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6036.787772] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6037.016479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6039.522417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6039.523708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.536356] LustreError: Skipped 1 previous similar message [ 6039.588521] Lustre: Skipped 3 previous similar messages [ 6053.343165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6053.570861] Lustre: server umount lustre-OST0000 complete [ 6060.903638] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6065.674423] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6095.666626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6102.219597] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6105.382957] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.414352] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6105.415197] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6105.443611] Lustre: Skipped 1 previous similar message [ 6110.166794] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.789562] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6120.928229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6120.934744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6120.940709] LustreError: Skipped 1 previous similar message [ 6120.951621] Lustre: Skipped 14 previous similar messages [ 6122.346684] Lustre: server umount lustre-OST0000 complete [ 6140.383621] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6140.877931] Lustre: server umount lustre-MDT0000 complete [ 6141.411384] LustreError: 212702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.433708] LustreError: 212702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6146.785854] LustreError: 211954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277581 with bad export cookie 16983664832004392070 [ 6146.798347] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.824895] LustreError: 211954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6147.486612] Lustre: server umount lustre-MDT0001 complete [ 6159.598659] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:46:32 (1788277592) [ 6166.635677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6171.710765] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6180.465261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.138951] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6195.396993] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6202.608716] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6210.231339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.730945] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6228.806229] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6233.750627] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6233.996394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6239.129849] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6239.447774] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6246.896282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6247.177957] Lustre: lustre-OST0001: new disk, initializing [ 6247.209644] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6247.217741] Lustre: 221319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6254.381883] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6254.782942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6254.803034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6254.953121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6265.490751] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6270.944126] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6271.202362] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6275.822523] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6276.074717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6280.687501] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6280.693254] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6280.693265] Lustre: Skipped 4 previous similar messages [ 6284.169496] Lustre: server umount lustre-OST0001 complete [ 6290.811618] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6291.004898] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6294.911778] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6295.168415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6324.608619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6324.833397] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6324.854266] Lustre: Skipped 5 previous similar messages [ 6329.848017] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6329.860200] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6329.862385] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6330.374579] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6338.038561] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6340.201573] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6348.379736] Lustre: server umount lustre-OST0001 complete [ 6371.267874] Lustre: server umount lustre-OST0000 complete [ 6380.517847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6380.526154] Lustre: Skipped 16 previous similar messages [ 6381.990672] Lustre: server umount lustre-MDT0000 complete [ 6386.147415] LustreError: 217928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277820 with bad export cookie 16983664832004393029 [ 6386.157554] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6386.160000] LustreError: 217928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6392.775689] Lustre: server umount lustre-MDT0001 complete [ 6418.797044] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:50:51 (1788277851) [ 6427.473149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6427.992909] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6428.048080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6433.394418] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6444.566698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6444.655325] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6444.680550] Lustre: 226561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6450.277584] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6459.574817] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6466.124159] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6473.659963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6473.947780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6480.907091] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6489.865509] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6501.307760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.452647] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6508.114409] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6518.136882] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.722633] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6526.111990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6526.368828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6531.871485] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6532.104717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6539.209771] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a66c4725000.ost_server_uuid 50 [ 6541.178458] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9a66c4725000.ost_server_uuid in IDLE state after 0 sec [ 6542.220777] Lustre: Permanently deactivating lustre-OST0001 [ 6542.234710] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6546.660937] Lustre: Permanently reactivating lustre-OST0001 [ 6546.680386] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6546.685314] Lustre: Skipped 2 previous similar messages [ 6552.048034] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6552.066123] LustreError: Skipped 5 previous similar messages [ 6552.082291] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6552.103860] Lustre: Skipped 7 previous similar messages [ 6558.131438] Lustre: server umount lustre-OST0001 complete [ 6561.765301] LustreError: 228407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6561.803395] LustreError: 228407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6570.532743] Lustre: server umount lustre-OST0000 complete [ 6582.092791] Lustre: server umount lustre-MDT0000 complete [ 6585.964373] LustreError: 226555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278020 with bad export cookie 16983664832004394324 [ 6585.969638] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6585.976940] LustreError: 226555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6586.369296] Lustre: server umount lustre-MDT0001 complete [ 6611.291326] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:54:04 (1788278044) [ 6618.726079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6618.978773] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6619.017596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6619.040084] Lustre: Skipped 2 previous similar messages [ 6624.287622] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6632.863754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.994147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6633.025602] Lustre: 233258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6637.556335] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6644.575305] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6650.839507] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6659.262234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.444260] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6659.466806] Lustre: Permanently deactivating lustre-OST0000 [ 6659.485791] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6665.790580] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6667.135724] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6676.057399] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6683.234496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6683.321367] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6689.288529] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6692.098447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6697.801842] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6699.680499] Lustre: Permanently reactivating lustre-OST0000 [ 6699.732743] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6699.744840] Lustre: Skipped 2 previous similar messages [ 6700.646048] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6700.651508] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 000000001c6ec215, cur 1788278135 deadline 1788278130 last 1788278100 [ 6700.662859] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6700.694065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6700.701563] Lustre: Skipped 1 previous similar message [ 6700.746554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6714.876969] Lustre: server umount lustre-OST0001 complete [ 6724.702049] Lustre: server umount lustre-OST0000 complete [ 6734.677433] Lustre: server umount lustre-MDT0000 complete [ 6738.724251] LustreError: 235127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278173 with bad export cookie 16983664832004395969 [ 6738.739226] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.740218] LustreError: 235127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6739.198424] Lustre: server umount lustre-MDT0001 complete [ 6752.977105] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 6756.675676] Key type lgssc unregistered [ 6757.059697] LNet: 238142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6757.070816] LNetError: 238142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6757.096028] LNet: Removed LNI 192.168.203.154@tcp [ 6757.802510] Key type .llcrypt unregistered [ 6757.804600] Key type ._llcrypt unregistered [ 6782.313085] Key type ._llcrypt registered [ 6782.314739] Key type .llcrypt registered [ 6782.416990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 6783.693611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6783.720315] alg: No test for adler32 (adler32-zlib) [ 6784.790849] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 6785.055058] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 6786.759176] Key type lgssc registered [ 6787.999678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6799.049389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6799.066353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6800.491238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6804.721510] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6810.085044] LustreError: 239612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.113786] LustreError: 239612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6814.270139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.374843] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6814.402198] Lustre: 239601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6814.417527] Lustre: Permanently deactivating lustre-MDT0001 [ 6814.430566] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6814.674176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6815.703990] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6815.710201] LustreError: 239643:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6815.720290] LustreError: 239643:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6815.739472] LustreError: 239643:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6819.484418] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6828.684919] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6834.926684] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6842.409367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6842.756985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6844.771361] LustreError: 241454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6844.783275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6850.033742] LustreError: 241454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6850.193986] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6855.139579] LustreError: 241454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6859.100868] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6860.271900] LustreError: 241452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6860.282664] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6860.297349] LustreError: 241452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6866.916520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6867.121062] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6872.574379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6872.581584] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6873.978653] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 6882.463917] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6886.534361] Lustre: Permanently reactivating lustre-MDT0001 [ 6886.550703] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6886.559761] Lustre: Skipped 2 previous similar messages [ 6892.155355] Lustre: lustre-MDT0001: Client b4aedf60-7d22-4c86-b943-70c9090a5bf0 (at 192.168.203.54@tcp) reconnecting [ 6899.022605] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6900.854183] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6906.956592] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6907.161511] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6911.868460] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6912.133293] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6914.867894] Lustre: Permanently deactivating lustre-MDT0001 [ 6914.882958] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6914.889594] Lustre: Skipped 2 previous similar messages [ 6924.433300] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6931.488207] Lustre: server umount lustre-MDT0000 complete [ 6932.450259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6932.455881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.465636] LustreError: 240356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6932.479430] LustreError: 240356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6933.986639] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6934.013720] Lustre: Skipped 2 previous similar messages [ 6935.163482] LustreError: 239594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278369 with bad export cookie 17285989391557896583 [ 6935.173786] LustreError: 239594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6935.180968] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6935.493295] Lustre: server umount lustre-MDT0001 complete [ 6941.751353] Lustre: server umount lustre-OST0000 complete [ 6954.228210] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:59:47 (1788278387) [ 6965.671786] Lustre: server umount lustre-OST0001 complete [ 6971.533054] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 6980.359423] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7017.549180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.811224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7017.821296] Lustre: Skipped 2 previous similar messages [ 7017.857052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7017.929799] Lustre: lustre-MDT0000: new disk, initializing [ 7017.979894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.998536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7022.344964] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7033.792209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.919592] Lustre: 247866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7033.995086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7034.001329] Lustre: Skipped 1 previous similar message [ 7034.136706] Lustre: lustre-MDT0001: new disk, initializing [ 7034.283246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.345582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7034.366952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7039.064614] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7049.659448] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7055.618057] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7063.327891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7063.674506] Lustre: lustre-OST0000: new disk, initializing [ 7063.680765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7063.690936] Lustre: 249847:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7063.847247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.700295] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7071.838067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7071.852758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7072.080191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7081.737607] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.699422] LustreError: 247876:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7092.676538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7092.940383] Lustre: lustre-OST0001: new disk, initializing [ 7092.945805] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7092.960694] Lustre: 250966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7093.713339] LustreError: 247876:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7093.735883] LustreError: 250060:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7099.663297] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7103.071364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7103.081659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7103.173971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7103.736441] LustreError: 250060:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7108.812574] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7110.609973] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7113.701667] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.703707] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.727963] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7113.735753] Lustre: Skipped 1 previous similar message [ 7114.865620] Lustre: lustre-OST0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 7118.602661] Lustre: server umount lustre-OST0001 complete [ 7118.833063] LustreError: 249832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.848787] LustreError: 249832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7138.799587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7138.800807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7138.815686] LustreError: Skipped 2 previous similar messages [ 7138.816962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7138.859411] Lustre: Skipped 2 previous similar messages [ 7140.856736] Lustre: server umount lustre-OST0000 complete [ 7144.424186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.448746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7144.452159] Lustre: Skipped 4 previous similar messages [ 7149.540433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.546242] Lustre: Skipped 1 previous similar message [ 7150.748810] Lustre: server umount lustre-MDT0000 complete [ 7154.195663] LustreError: 247859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278588 with bad export cookie 17285989391557899628 [ 7154.200582] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7154.220368] LustreError: 247859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7160.818257] Lustre: server umount lustre-MDT0001 complete [ 7169.880095] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7172.129889] Key type lgssc unregistered [ 7172.364893] LNet: 253000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.375476] LNetError: 253000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7172.397148] LNet: Removed LNI 192.168.203.154@tcp [ 7173.094803] Key type .llcrypt unregistered [ 7173.096991] Key type ._llcrypt unregistered [ 7208.124797] Key type ._llcrypt registered [ 7208.128242] Key type .llcrypt registered [ 7208.230501] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7209.290349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7209.429924] alg: No test for adler32 (adler32-zlib) [ 7210.605702] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 7210.888511] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7212.671233] Key type lgssc registered [ 7214.346792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7226.793818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7226.828477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.347798] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7228.409821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7228.714050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.764286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7243.177079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.337629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7243.377963] Lustre: 255209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7243.548931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7249.161543] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7258.110187] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7263.578898] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.373519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7270.602413] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7270.886290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7277.260150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7277.670848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7285.201547] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7338.468711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.481056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.501947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7338.982820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7338.996798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7339.020407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7343.069347] Lustre: server umount lustre-OST0000 complete [ 7347.649642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7363.925316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7364.371785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7372.308360] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7378.732682] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7378.750421] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7378.756260] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7378.774491] Lustre: Skipped 1 previous similar message [ 7380.796645] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7424.995274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7424.998178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7425.011630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7425.034668] Lustre: Skipped 1 previous similar message [ 7426.557565] Lustre: server umount lustre-OST0000 complete [ 7433.184619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7433.203373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.221441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.229039] Lustre: Skipped 1 previous similar message [ 7437.093539] Lustre: server umount lustre-MDT0000 complete [ 7440.869365] LustreError: 255202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278875 with bad export cookie 17418585658369437621 [ 7440.875434] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7440.880714] LustreError: 255202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7441.218493] Lustre: server umount lustre-MDT0001 complete [ 7452.355414] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7455.381044] Key type lgssc unregistered [ 7455.714066] LNet: 260639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7455.727970] LNetError: 260639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7455.756335] LNet: Removed LNI 192.168.203.154@tcp [ 7456.752948] Key type .llcrypt unregistered [ 7456.756550] Key type ._llcrypt unregistered [ 7482.777504] Key type ._llcrypt registered [ 7482.780443] Key type .llcrypt registered [ 7482.901873] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7483.979369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7484.060879] alg: No test for adler32 (adler32-zlib) [ 7485.292251] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 7485.569583] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7487.279319] Key type lgssc registered [ 7488.335731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7496.448294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7496.471254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.942876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7502.702634] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7508.454615] LustreError: 262060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.485673] LustreError: 262060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7511.260753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.763737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7516.743700] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7524.779477] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7530.937948] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7538.446346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7538.930107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.023855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7544.855688] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7554.266098] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7584.224843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7584.235386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7584.247972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.325189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7588.326668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7588.340652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.866715] Lustre: server umount lustre-OST0000 complete [ 7593.440604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.443369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.465501] Lustre: Skipped 1 previous similar message [ 7593.487515] Lustre: Skipped 2 previous similar messages [ 7598.562661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7598.574869] Lustre: Skipped 1 previous similar message [ 7599.486035] Lustre: server umount lustre-MDT0000 complete [ 7603.384130] LustreError: 265327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279037 with bad export cookie 17999433687839844716 [ 7603.390084] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.394039] LustreError: 265327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7603.768189] Lustre: server umount lustre-MDT0001 complete [ 7613.616657] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7616.443134] Key type lgssc unregistered [ 7616.749648] LNet: 266369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7616.765731] LNetError: 266369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7616.796685] LNet: Removed LNI 192.168.203.154@tcp [ 7617.588739] Key type .llcrypt unregistered [ 7617.591238] Key type ._llcrypt unregistered [ 7635.487343] Key type ._llcrypt registered [ 7635.490831] Key type .llcrypt registered [ 7635.623565] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7636.688281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7636.724965] alg: No test for adler32 (adler32-zlib) [ 7637.701938] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 7637.908323] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7639.633147] Key type lgssc registered [ 7641.140920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7650.290380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7650.315307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.980754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7656.699442] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7661.538256] LustreError: 267592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.563978] LustreError: 267592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7665.749674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7666.002965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7670.286837] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7677.235530] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7682.800269] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7690.724790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7690.982614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.182926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7697.626512] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7706.043459] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7721.952185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.959699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.966649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.986046] Lustre: Skipped 1 previous similar message [ 7726.166156] Lustre: server umount lustre-OST0000 complete [ 7732.195602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.212714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.227112] Lustre: Skipped 1 previous similar message [ 7732.244923] Lustre: Skipped 2 previous similar messages [ 7736.322587] Lustre: server umount lustre-MDT0000 complete [ 7737.313590] LustreError: 267591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.347745] LustreError: 267591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7740.601372] LustreError: 267572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279175 with bad export cookie 18280018313231366653 [ 7740.607402] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7740.613380] LustreError: 267572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7741.069332] Lustre: server umount lustre-MDT0001 complete [ 7753.254501] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7756.368743] Key type lgssc unregistered [ 7756.682503] LNet: 271272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7756.692259] LNetError: 271272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7756.705349] LNet: Removed LNI 192.168.203.154@tcp [ 7757.612992] Key type .llcrypt unregistered [ 7757.614763] Key type ._llcrypt unregistered [ 7783.422112] Key type ._llcrypt registered [ 7783.424532] Key type .llcrypt registered [ 7783.523175] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7784.554277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7784.577762] alg: No test for adler32 (adler32-zlib) [ 7785.661911] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 7785.927345] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7787.607176] Key type lgssc registered [ 7788.759797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7796.112711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7796.129384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.515303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.422227] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7806.945833] LustreError: 272693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.958831] LustreError: 272693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7810.163396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.458405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7814.544244] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7821.671767] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7827.299293] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7835.244721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7835.649803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7841.579882] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7841.783628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7849.666529] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7880.685927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.697065] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7880.706715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7884.762631] Lustre: server umount lustre-OST0000 complete [ 7890.928279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.941784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7890.943581] Lustre: Skipped 2 previous similar messages [ 7890.956771] Lustre: Skipped 2 previous similar messages [ 7896.036210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.048652] Lustre: Skipped 1 previous similar message [ 7901.156166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7901.168402] Lustre: Skipped 1 previous similar message [ 7903.199180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7903.432290] Lustre: server umount lustre-MDT0000 complete [ 7906.273757] LustreError: 272689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.309457] LustreError: 272689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7908.005325] LustreError: 272675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279342 with bad export cookie 14083711193286464621 [ 7908.014197] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7908.025205] LustreError: 272675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7908.609909] Lustre: server umount lustre-MDT0001 complete [ 7919.581580] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 7922.910411] Key type lgssc unregistered [ 7923.231792] LNet: 277005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7923.238802] LNetError: 277005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7923.258061] LNet: Removed LNI 192.168.203.154@tcp [ 7924.226190] Key type .llcrypt unregistered [ 7924.228599] Key type ._llcrypt unregistered [ 7941.953602] Key type ._llcrypt registered [ 7941.955205] Key type .llcrypt registered [ 7942.106922] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 7942.971379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7942.997643] alg: No test for adler32 (adler32-zlib) [ 7944.265272] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 7944.590258] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 7946.391351] Key type lgssc registered [ 7947.725197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7957.048714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7957.064708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.791778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.128108] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7969.267152] LustreError: 278244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7969.310341] LustreError: 278244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7973.336316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7973.645410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7978.013721] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 7985.580856] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7990.975809] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.038342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.365213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.551489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8005.027758] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8013.353951] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8030.177019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8030.188569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.191791] LustreError: Skipped 1 previous similar message [ 8030.192661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8030.244575] Lustre: Skipped 1 previous similar message [ 8032.977731] Lustre: server umount lustre-OST0000 complete [ 8040.416472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8040.418392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8040.438702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8040.466134] Lustre: Skipped 2 previous similar messages [ 8043.764953] Lustre: server umount lustre-MDT0000 complete [ 8045.538319] LustreError: 278224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8045.557726] LustreError: 278224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8047.757408] LustreError: 278208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279482 with bad export cookie 1808295601190263444 [ 8047.760165] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8047.777760] LustreError: 278208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8048.419369] Lustre: server umount lustre-MDT0001 complete [ 8063.434972] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8067.714370] Key type lgssc unregistered [ 8068.085249] LNet: 281948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8068.093370] LNetError: 281948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8068.116523] LNet: Removed LNI 192.168.203.154@tcp [ 8068.819627] Key type .llcrypt unregistered [ 8068.821251] Key type ._llcrypt unregistered [ 8083.292864] Key type ._llcrypt registered [ 8083.296952] Key type .llcrypt registered [ 8083.467980] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8101.827949] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8121.550451] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8123.288401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8123.357319] alg: No test for adler32 (adler32-zlib) [ 8124.502761] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 8124.782963] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8126.463527] Key type lgssc registered [ 8127.669784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8167.461245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8167.492627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.816415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8168.868974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8169.008241] Lustre: lustre-MDT0000: new disk, initializing [ 8169.105802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.127129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8174.001739] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8186.910877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8187.182566] Lustre: 285502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8187.287820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8187.299553] Lustre: Skipped 1 previous similar message [ 8187.472685] Lustre: lustre-MDT0001: new disk, initializing [ 8187.597052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8187.656280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8187.671429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8193.629133] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8205.336754] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8211.968364] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8219.358474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8219.659657] Lustre: lustre-OST0000: new disk, initializing [ 8219.664503] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8219.674529] Lustre: 287472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8219.751509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8226.675379] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8227.950991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8227.973190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8228.187523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8236.959944] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8242.476664] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8242.678244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8248.148786] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8248.470495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8253.920910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8253.949161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.979600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8256.657044] Lustre: server umount lustre-OST0000 complete [ 8264.162076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8264.169578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8264.192515] Lustre: Skipped 2 previous similar messages [ 8264.224398] Lustre: Skipped 2 previous similar messages [ 8267.022008] Lustre: server umount lustre-MDT0000 complete [ 8271.154734] LustreError: 285494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279705 with bad export cookie 14486402816805446756 [ 8271.160128] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8271.163467] LustreError: 285494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8271.599366] Lustre: server umount lustre-MDT0001 complete [ 8281.158841] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:21:54 (1788279714) [ 8287.984752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.400887] LustreError: 289886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8288.418577] LustreError: 289886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8288.462948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8292.976195] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8293.860525] LustreError: 289887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.953610] LustreError: 289886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.611191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.095387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8308.179696] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8316.605871] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8322.628976] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8329.700453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.153235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8337.116947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8345.359124] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8357.869554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.886637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8357.891694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8357.903608] LustreError: Skipped 1 previous similar message [ 8362.043583] Lustre: server umount lustre-OST0000 complete [ 8368.100627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8368.101642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8368.120106] Lustre: Skipped 2 previous similar messages [ 8368.134475] Lustre: Skipped 2 previous similar messages [ 8373.216110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8373.226796] Lustre: Skipped 1 previous similar message [ 8380.383612] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8380.669786] Lustre: server umount lustre-MDT0000 complete [ 8383.459182] LustreError: 289886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8383.475965] LustreError: 289886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8384.834861] LustreError: 289868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279819 with bad export cookie 14486402816805447638 [ 8384.848558] LustreError: 289868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8384.850562] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8385.256886] Lustre: server umount lustre-MDT0001 complete [ 8395.754947] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8398.710475] Key type lgssc unregistered [ 8399.041638] LNet: 293424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8399.054954] LNetError: 293424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8400.108610] LNet: Removed LNI 192.168.203.154@tcp [ 8401.002648] Key type .llcrypt unregistered [ 8401.006207] Key type ._llcrypt unregistered [ 8428.535945] Key type ._llcrypt registered [ 8428.539320] Key type .llcrypt registered [ 8428.669419] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8443.950749] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8444.914517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8444.971034] alg: No test for adler32 (adler32-zlib) [ 8446.033233] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 8446.277031] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8447.935360] Key type lgssc registered [ 8448.932065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8493.067564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8493.085673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.314630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8494.338553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8494.393490] Lustre: lustre-MDT0000: new disk, initializing [ 8494.456469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8494.469756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8498.878665] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8509.521409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.657503] Lustre: 297251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8509.703947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8509.707176] Lustre: Skipped 1 previous similar message [ 8509.773859] Lustre: lustre-MDT0001: new disk, initializing [ 8509.906354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8509.947477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8509.963159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8514.302541] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8523.497991] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8529.361691] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8535.699984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8535.979730] Lustre: lustre-OST2710: new disk, initializing [ 8535.992763] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8536.001502] Lustre: 299218:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8536.104865] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8541.278119] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8543.778290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8543.789842] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8543.891773] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8550.240978] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8556.134956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8556.325706] Lustre: lustre-OST03e8: new disk, initializing [ 8556.331797] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8556.337382] Lustre: 300283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8556.422580] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8561.731153] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8563.288585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8563.304746] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8563.364382] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8568.826122] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8575.484785] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8575.704570] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8579.810916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8580.006843] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8584.166509] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8584.359646] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8587.489369] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8587.674793] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8591.847478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8591.854609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.878932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8594.402323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8594.404390] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8598.483131] Lustre: server umount lustre-MDT0000 complete [ 8602.227111] LustreError: 300281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280036 with bad export cookie 15601466255803351138 [ 8602.230381] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8602.238581] LustreError: 300281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8602.544751] Lustre: server umount lustre-MDT0001 complete [ 8617.154936] Lustre: server umount lustre-OST2710 complete [ 8631.452727] Lustre: server umount lustre-OST03e8 complete [ 8648.713552] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 8657.293169] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8689.127438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8689.368526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8689.391896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8689.492573] Lustre: lustre-MDT0000: new disk, initializing [ 8689.577071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8689.609208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8693.680710] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8703.736429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.861373] Lustre: 305531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8703.907168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8703.911150] Lustre: Skipped 1 previous similar message [ 8703.973417] Lustre: lustre-MDT0001: new disk, initializing [ 8704.080346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8704.112475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8704.127067] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8708.272825] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8716.605077] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8721.539492] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8727.550242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8727.816527] Lustre: lustre-OST0000: new disk, initializing [ 8727.820951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8727.832307] Lustre: 307500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8727.934168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.587334] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8737.841568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8737.854785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8737.974659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8743.387888] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8748.094375] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8748.270377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8753.059803] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8753.338353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8758.759508] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.760233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.764869] LustreError: Skipped 1 previous similar message [ 8758.767815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.782588] Lustre: Skipped 3 previous similar messages [ 8758.792844] Lustre: Skipped 3 previous similar messages [ 8761.424457] Lustre: server umount lustre-OST0000 complete [ 8765.419705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.446237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.473867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.476741] Lustre: Skipped 1 previous similar message [ 8770.959931] Lustre: server umount lustre-MDT0000 complete [ 8774.116565] LustreError: 307786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.140259] LustreError: 307786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8774.541726] LustreError: 305524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280209 with bad export cookie 15601466255803352818 [ 8774.547428] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8774.560264] LustreError: 305524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8775.087821] Lustre: server umount lustre-MDT0001 complete [ 8782.987691] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:30:16 (1788280216) [ 8784.245327] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8785.728263] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:30:19 (1788280219) [ 8789.177133] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8815.503318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.712066] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8815.754227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8815.917392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.923802] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8828.385905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8828.537428] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8828.596400] Lustre: 311355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8834.261227] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8856.345433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8856.748209] Lustre: Denying initial registration attempt from nid 192.168.203.154@tcp, specified as failover [ 8856.756560] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8856.765527] LustreError: 312808:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8856.774392] LustreError: 312808:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8856.785225] LustreError: 312808:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8856.915385] Lustre: server umount lustre-OST0000 complete [ 8856.919820] LustreError: 312808:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8868.326231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.336526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.337694] Lustre: Skipped 2 previous similar messages [ 8868.370779] Lustre: Skipped 3 previous similar messages [ 8871.252669] Lustre: server umount lustre-MDT0000 complete [ 8873.441766] LustreError: 311362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8873.457877] LustreError: 311362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8874.873594] LustreError: 311346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280309 with bad export cookie 15601466255803353700 [ 8874.886993] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8874.892769] LustreError: 311346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8875.359164] Lustre: server umount lustre-MDT0001 complete [ 8885.424824] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 8888.274768] Key type lgssc unregistered [ 8888.543266] LNet: 313932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8888.563889] LNetError: 313932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8888.590554] LNet: Removed LNI 192.168.203.154@tcp [ 8889.414525] Key type .llcrypt unregistered [ 8889.416891] Key type ._llcrypt unregistered [ 8899.376572] Key type ._llcrypt registered [ 8899.378920] Key type .llcrypt registered [ 8899.461527] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8900.324318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8900.460161] alg: No test for adler32 (adler32-zlib) [ 8901.604941] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 8901.837077] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 8903.527360] Key type lgssc registered [ 8904.662228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.724876] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 8954.290368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8954.318276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.546502] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8955.566714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8955.766727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8959.749971] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8967.696290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.777104] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8967.804408] Lustre: 316696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8968.058500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8972.535665] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 8979.204705] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8984.080851] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9003.066192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9003.266952] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9003.488714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9010.476557] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9018.672963] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9023.457471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9023.465614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.479507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9023.971103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9023.983407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.995435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9026.789391] Lustre: server umount lustre-OST0000 complete [ 9034.217269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9034.223136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9034.242264] Lustre: Skipped 1 previous similar message [ 9034.275131] Lustre: Skipped 1 previous similar message [ 9037.405784] Lustre: server umount lustre-MDT0000 complete [ 9041.927328] LustreError: 316688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280476 with bad export cookie 5472752384356028863 [ 9041.934175] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9041.943141] LustreError: 316688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9042.596461] Lustre: server umount lustre-MDT0001 complete [ 9051.428245] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9053.953290] Key type lgssc unregistered [ 9054.236347] LNet: 320271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9054.248829] LNetError: 320271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9054.264440] LNet: Removed LNI 192.168.203.154@tcp [ 9054.896201] Key type .llcrypt unregistered [ 9054.897752] Key type ._llcrypt unregistered [ 9075.385921] Key type ._llcrypt registered [ 9075.389139] Key type .llcrypt registered [ 9075.485751] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9076.094498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9076.209586] alg: No test for adler32 (adler32-zlib) [ 9077.228110] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 9077.444982] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9079.079211] Key type lgssc registered [ 9079.977486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9086.988220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9087.000994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9088.410363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9092.056731] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9097.702147] LustreError: 321668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.726945] LustreError: 321668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9099.977832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.370076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9105.241393] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9113.645612] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9119.125210] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9127.723262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9128.038031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.465617] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9140.496080] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9154.114762] Lustre: server umount lustre-MDT0000 complete [ 9156.066590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9156.067977] LustreError: 321668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9156.074827] Lustre: Skipped 1 previous similar message [ 9157.077391] LustreError: 321648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280591 with bad export cookie 10695131229736673392 [ 9157.085112] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9158.766093] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [ 9158.781162] LustreError: 321663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.792565] LustreError: 321663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9161.185640] LustreError: 321665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.186853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9161.232530] Lustre: Skipped 1 previous similar message [ 9161.235857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9163.447387] Lustre: server umount lustre-MDT0001 complete [ 9169.015633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9175.812564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9181.880389] LustreError: 323499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.905869] LustreError: 323499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9181.939523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9181.984458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9185.703761] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9192.172032] LustreError: 325281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9192.177714] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9192.186774] LustreError: 325281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9193.004860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9193.462049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9198.449157] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9198.582669] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9206.526731] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9208.479908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9214.006599] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.409672] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9223.140562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9223.146642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.164623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9223.173228] Lustre: Skipped 1 previous similar message [ 9224.168122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9226.586865] Lustre: server umount lustre-OST0000 complete [ 9234.405359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.414366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9234.429332] Lustre: Skipped 2 previous similar messages [ 9234.440334] Lustre: Skipped 3 previous similar messages [ 9236.565307] Lustre: server umount lustre-MDT0000 complete [ 9239.537483] LustreError: 325286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.561244] LustreError: 325286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9240.411694] LustreError: 321649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280674 with bad export cookie 10695131229736680525 [ 9240.414365] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9240.424809] LustreError: 321649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9240.855144] Lustre: server umount lustre-MDT0001 complete [ 9250.583094] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9253.002611] Key type lgssc unregistered [ 9253.279775] LNet: 327957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9253.295708] LNetError: 327957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9254.315774] LNet: Removed LNI 192.168.203.154@tcp [ 9255.315391] Key type .llcrypt unregistered [ 9255.318146] Key type ._llcrypt unregistered [ 9278.609861] Key type ._llcrypt registered [ 9278.612145] Key type .llcrypt registered [ 9278.697357] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9279.615298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9279.641747] alg: No test for adler32 (adler32-zlib) [ 9280.708848] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 9281.024807] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9282.743165] Key type lgssc registered [ 9283.844240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9291.031621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9291.077039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9292.518778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.797739] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9301.991874] LustreError: 329391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9302.006499] LustreError: 329391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9304.352527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9304.654836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9308.997986] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9316.444617] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9322.561996] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9327.076037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9327.084836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9327.086571] Lustre: Skipped 1 previous similar message [ 9327.097653] Lustre: Skipped 1 previous similar message [ 9330.150766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9330.157666] Lustre: Skipped 1 previous similar message [ 9331.637775] Lustre: server umount lustre-MDT0000 complete [ 9334.889176] LustreError: 329354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280769 with bad export cookie 13126225345550352894 [ 9334.902614] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9334.903693] LustreError: 329354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9335.297836] Lustre: server umount lustre-MDT0001 complete [ 9350.153268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9350.423264] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9350.452124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9350.618783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9355.160640] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9363.046787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.152344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9363.202390] Lustre: 331961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9363.452666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9367.423420] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9374.772035] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9380.321621] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9391.470842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.753735] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9391.763800] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9391.770215] LustreError: 333846:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9391.779409] LustreError: 333846:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9391.785743] LustreError: 333846:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9391.907403] Lustre: server umount lustre-OST0000 complete [ 9391.914265] LustreError: 333846:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9402.422862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9402.661555] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9402.801230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9408.763922] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9409.792322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9415.569787] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9422.948724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9423.042804] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9423.108679] Lustre: lustre-OST0001: new disk, initializing [ 9423.112190] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9423.123637] Lustre: 335388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9423.231027] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9428.591169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9428.602216] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9428.704088] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9428.936766] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9436.967538] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9444.321844] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9444.336959] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9444.354796] Lustre: Skipped 1 previous similar message [ 9444.360552] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9444.373047] Lustre: Skipped 1 previous similar message [ 9446.747712] Lustre: server umount lustre-OST0001 complete [ 9449.453927] LustreError: 334311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9449.498541] LustreError: 334311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9454.562463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9454.563326] LustreError: 334314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9454.576513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9454.588891] Lustre: Skipped 1 previous similar message [ 9454.612300] LustreError: 334314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9459.683113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9459.688438] LustreError: 334314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9459.694527] Lustre: Skipped 2 previous similar messages [ 9459.732556] LustreError: 334314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9464.821247] LustreError: 336306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.836339] LustreError: 336306:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9466.335918] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9466.570879] Lustre: server umount lustre-OST0000 complete [ 9470.947316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9470.951134] LustreError: Skipped 1 previous similar message [ 9470.955746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.968698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9470.976877] Lustre: Skipped 2 previous similar messages [ 9475.040909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9476.488936] Lustre: server umount lustre-MDT0000 complete [ 9480.168375] LustreError: 331974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9480.184708] LustreError: 331974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9480.461608] LustreError: 331953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280914 with bad export cookie 13126225345550353447 [ 9480.461608] LustreError: 331954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280914 with bad export cookie 13126225345550353447 [ 9480.463209] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9480.471628] LustreError: 331954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9480.901464] Lustre: server umount lustre-MDT0001 complete [ 9491.861671] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [ 9494.955419] Key type lgssc unregistered [ 9495.312583] LNet: 337354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9495.340611] LNetError: 337354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9496.361477] LNet: Removed LNI 192.168.203.154@tcp [ 9497.241662] Key type .llcrypt unregistered [ 9497.244544] Key type ._llcrypt unregistered [ 9566.104907] Key type ._llcrypt registered [ 9566.106230] Key type .llcrypt registered [ 9566.178868] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [ 9582.694944] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [ 9584.075792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9584.098691] alg: No test for adler32 (adler32-zlib) [ 9585.238145] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [ 9585.576225] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [ 9587.279193] Key type lgssc registered [ 9589.066691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9625.011398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9625.067918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9626.402371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9626.437901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9626.541636] Lustre: lustre-MDT0000: new disk, initializing [ 9626.651191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9626.682210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9631.531725] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9641.575198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9641.695990] Lustre: 342289:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9641.778941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9641.783293] Lustre: Skipped 1 previous similar message [ 9641.885908] Lustre: lustre-MDT0001: new disk, initializing [ 9641.995158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9642.048130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9642.065479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9646.750729] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9656.082161] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9661.791878] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9668.030456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9668.338177] Lustre: lustre-OST0000: new disk, initializing [ 9668.345747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9668.356453] Lustre: 344260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9668.457600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.371686] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9676.847421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9676.862690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9676.899411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9683.165691] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9687.754759] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9687.998812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9692.275416] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9692.437243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9697.253909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9697.266110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.274976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9697.296833] Lustre: Skipped 1 previous similar message [ 9697.307245] Lustre: Skipped 1 previous similar message [ 9700.292498] Lustre: server umount lustre-OST0000 complete [ 9707.499555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.507680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9707.515541] Lustre: Skipped 1 previous similar message [ 9707.526495] Lustre: Skipped 1 previous similar message [ 9709.923955] Lustre: server umount lustre-MDT0000 complete [ 9712.864283] LustreError: 342279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281147 with bad export cookie 11766535894117984322 [ 9712.876156] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9712.878563] LustreError: 342279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9713.292983] Lustre: server umount lustre-MDT0001 complete [ 9721.039617] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:45:54 (1788281154) [ 9728.678543] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:46:01 (1788281161) [ 9737.184154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.858959] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.874479] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9737.927770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.044240] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9743.330652] LustreError: 347016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.459777] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.484951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.808633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9755.442658] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9762.468986] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9767.858738] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9774.885418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9775.220618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9781.450467] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9789.260745] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9796.880220] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9798.754391] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9800.604939] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9802.637589] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9804.723509] Lustre: Failing over lustre-MDT0000 [ 9805.040783] Lustre: server umount lustre-MDT0000 complete [ 9806.453389] LustreError: 347012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.330379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9807.341084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9809.386935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.539671] LustreError: 347010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.553852] LustreError: 347010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9819.624373] LustreError: 348304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.658517] LustreError: 348304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9824.548169] Lustre: 340514:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281243/real 1788281243] req@000000002e255112 x1875148558898688/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788281259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9824.571905] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9824.673464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9834.985448] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa34b220dcdc92fd5 [ 9835.017894] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [ 9835.345573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9835.405917] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9838.167696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9840.648229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9840.683443] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9840.690284] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9840.740756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9851.165868] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9852.971150] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9857.252828] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9859.664132] Lustre: Failing over lustre-MDT0000 [ 9860.231861] Lustre: server umount lustre-MDT0000 complete [ 9861.094489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9861.096281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9861.104752] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9861.104764] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [ 9861.139846] Lustre: Skipped 2 previous similar messages [ 9877.345614] Lustre: 340514:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281295/real 1788281295] req@00000000e1bef154 x1875148558920064/t0(0) o400->MGC192.168.203.154@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788281311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9877.380962] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9879.622987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9886.688489] LustreError: 340510:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009594889a x1875148558925696/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9887.102787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9887.154535] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9890.390895] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9892.342393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9892.348816] Lustre: Skipped 2 previous similar messages [ 9892.383498] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 9892.433469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9892.522323] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9902.777892] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9904.381841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9912.801526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9912.807610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9912.810289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9912.844944] Lustre: Skipped 3 previous similar messages [ 9916.785787] Lustre: server umount lustre-MDT0000 complete [ 9917.922202] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9917.950309] LustreError: 347015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 9920.737283] LustreError: 349556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281355 with bad export cookie 11766535894117987080 [ 9920.744513] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9920.760026] LustreError: 349556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9921.344123] Lustre: server umount lustre-MDT0001 complete [ 9936.272974] Lustre: server umount lustre-OST0000 complete [ 9952.920444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9953.352557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9957.673711] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9967.681714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.340421] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [ 9981.624311] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9989.801122] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10000.658149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10009.299887] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10018.489350] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10022.903337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10030.405440] Lustre: server umount lustre-OST0000 complete [10034.656863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10034.666497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10034.687558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10034.693933] Lustre: Skipped 2 previous similar messages [10043.369318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10043.373737] Lustre: Skipped 3 previous similar messages [10044.537993] Lustre: server umount lustre-MDT0000 complete [10048.484318] LustreError: 353355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10048.518130] LustreError: 353355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10048.651535] LustreError: 353339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281483 with bad export cookie 11766535894117987801 [10048.663782] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10048.664831] LustreError: 353339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10059.419984] Lustre: server umount lustre-MDT0001 complete [10070.254772] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10073.083509] Key type lgssc unregistered [10073.355309] LNet: 356898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10073.367541] LNetError: 356898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10073.403561] LNet: Removed LNI 192.168.203.154@tcp [10074.139672] Key type .llcrypt unregistered [10074.142178] Key type ._llcrypt unregistered [10101.509747] Key type ._llcrypt registered [10101.512507] Key type .llcrypt registered [10101.605427] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10102.533964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10102.637340] alg: No test for adler32 (adler32-zlib) [10103.726900] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [10103.944745] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10105.671210] Key type lgssc registered [10106.678769] Lustre: Echo OBD driver; http://www.lustre.org/ [10114.277683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10114.323359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10115.759896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10120.213625] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10126.306351] LustreError: 358425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.313358] LustreError: 358425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10128.426552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.815396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10132.935870] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10142.172713] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10148.815512] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10155.628624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10155.977153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10161.151538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10162.515772] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10170.349703] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10189.803591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.819781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10189.828743] Lustre: Skipped 2 previous similar messages [10194.030640] Lustre: server umount lustre-MDT0000 complete [10194.919202] LustreError: 358421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10194.958477] LustreError: 358421:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10198.304549] LustreError: 359160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281632 with bad export cookie 5425601735932706839 [10198.311264] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10198.318502] LustreError: 359160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10198.812600] Lustre: server umount lustre-MDT0001 complete [10213.685925] Lustre: server umount lustre-OST0000 complete [10229.975176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.459095] LustreError: 362160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.587190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10235.881762] LustreError: 362161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10236.126357] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10239.985511] LustreError: 362160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10245.101308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10245.929956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10251.895526] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10260.994923] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10267.444384] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10274.980066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10275.438161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10282.466222] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10291.264362] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10300.022746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10317.795768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10317.809931] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10317.829730] Lustre: Skipped 2 previous similar messages [10317.835428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10318.824309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.846640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10318.859756] Lustre: Skipped 2 previous similar messages [10321.396956] Lustre: server umount lustre-MDT0000 complete [10323.938271] LustreError: 362161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.988643] LustreError: 362161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10325.858296] LustreError: 362140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281760 with bad export cookie 5425601735932707952 [10325.870915] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10325.881671] LustreError: 362140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10326.390595] Lustre: server umount lustre-MDT0001 complete [10341.222506] Lustre: server umount lustre-OST0000 complete [10357.816339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.320143] LustreError: 366095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.417050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.192720] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10372.867791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.092459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10377.357795] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10386.090891] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10392.548648] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10400.932661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10401.278588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10401.295515] Lustre: Skipped 1 previous similar message [10403.352229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10408.945267] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10418.823171] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10427.872726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10427.877485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.877519] Lustre: Skipped 1 previous similar message [10427.887289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.929977] Lustre: Skipped 1 previous similar message [10430.613566] Lustre: server umount lustre-OST0000 complete [10438.112880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.115254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.143622] Lustre: Skipped 2 previous similar messages [10438.159567] Lustre: Skipped 1 previous similar message [10440.923739] Lustre: server umount lustre-MDT0000 complete [10443.236197] LustreError: 366090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10443.256757] LustreError: 366090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10445.080749] LustreError: 366078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281879 with bad export cookie 5425601735932708967 [10445.094713] LustreError: 366078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10445.115986] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10445.517644] Lustre: server umount lustre-MDT0001 complete [10457.321327] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10460.599708] Key type lgssc unregistered [10460.915551] LNet: 369632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10460.929560] LNetError: 369632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10460.950909] LNet: Removed LNI 192.168.203.154@tcp [10461.900190] Key type .llcrypt unregistered [10461.905565] Key type ._llcrypt unregistered [10486.490304] Key type ._llcrypt registered [10486.492191] Key type .llcrypt registered [10486.659325] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10487.643846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10487.700614] alg: No test for adler32 (adler32-zlib) [10488.747638] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [10489.056753] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10490.857444] Key type lgssc registered [10492.392771] Lustre: Echo OBD driver; http://www.lustre.org/ [10502.977274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10502.999673] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10503.010482] LustreError: 371073:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10503.048255] LustreError: 371073:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10503.064377] LustreError: 371073:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10503.081617] LustreError: 371073:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10503.102519] LustreError: 371073:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10512.000894] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10512.011474] LustreError: 371472:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10512.036891] LustreError: 371472:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10512.046161] LustreError: 371472:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10512.054709] LustreError: 371472:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10512.067682] LustreError: 371472:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10531.738811] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10535.242878] Key type lgssc unregistered [10535.629630] LNet: 372477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10535.636142] LNetError: 372477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10535.654108] LNet: Removed LNI 192.168.203.154@tcp [10536.610064] Key type .llcrypt unregistered [10536.614430] Key type ._llcrypt unregistered [10558.033477] Key type ._llcrypt registered [10558.042994] Key type .llcrypt registered [10558.254647] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [10577.108120] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10578.575771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10578.614275] alg: No test for adler32 (adler32-zlib) [10579.691127] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [10579.901454] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10581.567657] Key type lgssc registered [10583.610501] Lustre: Echo OBD driver; http://www.lustre.org/ [10636.469139] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10636.519042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10637.887224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10637.935625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10638.053708] Lustre: lustre-MDT0000: new disk, initializing [10638.152779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10638.173662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10643.350286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10655.328560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.473891] Lustre: 375648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10655.536912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10655.541443] Lustre: Skipped 1 previous similar message [10655.612558] Lustre: lustre-MDT0001: new disk, initializing [10655.776785] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10655.818168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10655.838599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10661.100406] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10672.259264] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10679.602370] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10689.504110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10689.967510] Lustre: lustre-OST0000: new disk, initializing [10689.974773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10689.993806] Lustre: 377617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10690.164941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10696.290313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10696.306309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10696.512686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10697.846872] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10708.933128] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10716.349107] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10716.670435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10722.055965] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10722.424330] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10727.412805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10727.416377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10727.424706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10727.429284] LustreError: Skipped 1 previous similar message [10727.460600] Lustre: Skipped 1 previous similar message [10731.491278] Lustre: server umount lustre-OST0000 complete [10737.633353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10737.650146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10737.658030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10737.658039] Lustre: Skipped 1 previous similar message [10737.663818] Lustre: Skipped 1 previous similar message [10742.006617] Lustre: server umount lustre-MDT0000 complete [10745.740652] LustreError: 375639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282180 with bad export cookie 14669056136244637530 [10745.746958] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10745.758121] LustreError: 375639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10746.086693] Lustre: server umount lustre-MDT0001 complete [10755.751993] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10757.573189] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:03:10 (1788282190) [10764.797588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10765.352404] LustreError: 380079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.373870] LustreError: 380079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10765.464649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10770.302488] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10770.916433] LustreError: 380080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.018395] LustreError: 380079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.418516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.952761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10784.171200] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10791.505540] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10798.051204] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10804.152269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10804.504661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10810.722276] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10819.058959] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10827.409283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10827.698249] Lustre: lustre-OST0001: new disk, initializing [10827.713056] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10827.722550] Lustre: 382915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10827.860168] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10835.101565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10835.109623] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10835.259165] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10836.945639] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [10848.237507] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10855.916100] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10855.921650] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.938934] LustreError: Skipped 1 previous similar message [10855.942524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10855.967357] Lustre: Skipped 1 previous similar message [10855.988222] Lustre: Skipped 2 previous similar messages [10859.723444] Lustre: server umount lustre-OST0001 complete [10861.026287] LustreError: 381899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10861.040222] LustreError: 381899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10866.148670] LustreError: 381899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.162284] LustreError: 381899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10876.398847] LustreError: 383787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.425047] LustreError: 383787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10877.414483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10877.431509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10877.444533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10877.452888] Lustre: Skipped 1 previous similar message [10881.507027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.508543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10881.540691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10881.544898] Lustre: Skipped 1 previous similar message [10883.694377] Lustre: server umount lustre-OST0000 complete [10891.747193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10891.749866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10891.757322] Lustre: Skipped 1 previous similar message [10891.765626] Lustre: Skipped 1 previous similar message [10894.417303] Lustre: server umount lustre-MDT0000 complete [10896.867862] LustreError: 380080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.883594] LustreError: 380080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10897.980251] LustreError: 384154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282332 with bad export cookie 14669056136244638412 [10897.997487] LustreError: 384154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10898.002507] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.010684] LustreError: 373859:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [10904.472347] Lustre: server umount lustre-MDT0001 complete [10917.860296] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [10921.278517] Key type lgssc unregistered [10921.624434] LNet: 384910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10921.645552] LNetError: 384910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10921.664505] LNet: Removed LNI 192.168.203.154@tcp [10922.802295] Key type .llcrypt unregistered [10922.804707] Key type ._llcrypt unregistered [10981.666447] Key type ._llcrypt registered [10981.672038] Key type .llcrypt registered [10981.810620] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [10982.822125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10982.990674] alg: No test for adler32 (adler32-zlib) [10984.079862] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [10984.442507] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [10986.111162] Key type lgssc registered [10987.284823] Lustre: Echo OBD driver; http://www.lustre.org/ [10995.852753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10995.873640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.140504] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10997.188859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10997.381455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.079803] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11010.674984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.732564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11010.756352] Lustre: 387708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11010.950689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11015.263830] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11022.674718] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11029.336071] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11031.522289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11031.530197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11031.568238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11032.544184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11032.547227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11037.668191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11037.691403] Lustre: Skipped 2 previous similar messages [11037.864784] Lustre: server umount lustre-MDT0000 complete [11042.838825] LustreError: 387703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282477 with bad export cookie 2859107097081593069 [11042.878183] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11043.390772] Lustre: server umount lustre-MDT0001 complete [11056.332753] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:08:09 (1788282489) [11063.705577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.352622] LustreError: 390224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11064.368249] LustreError: 390224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11064.427789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11069.004454] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11069.929423] LustreError: 390225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.043652] LustreError: 390224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11077.780260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.137878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11082.699651] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11089.991358] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11095.803138] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.217742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11102.443874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11102.666270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11108.863623] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11117.213791] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11126.409383] Lustre: Permanently deactivating lustre-OST0000 [11126.430464] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11136.067561] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11138.742756] LustreError: 393211:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11138.769579] LustreError: 393211:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11138.778125] LustreError: 393211:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11143.100314] LustreError: 393260:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11143.114126] LustreError: 393260:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11153.544793] Lustre: server umount lustre-OST0000 complete [11156.696864] LustreError: 393510:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11156.701423] LustreError: 393510:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11160.045705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11160.050150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.061250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11166.177781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11166.182742] Lustre: Skipped 3 previous similar messages [11171.296979] LustreError: 391460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11171.341383] LustreError: 391460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11171.409448] Lustre: server umount lustre-MDT0000 complete [11175.336480] LustreError: 392057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282609 with bad export cookie 2859107097081593643 [11175.351617] LustreError: 392057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11175.364574] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11184.755714] Lustre: server umount lustre-MDT0001 complete [11195.130325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11199.848683] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11202.819239] LustreError: 394669:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11211.345955] Lustre: 395009:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.154@tcp [11214.103344] Lustre: server umount MGS complete [11228.671790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.292709] LustreError: 395734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11229.314409] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11229.321571] Lustre: Skipped 1 previous similar message [11229.434351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.594717] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11239.910425] LustreError: 395735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.944211] LustreError: 395735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11246.153243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11247.048853] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11247.226479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11253.182933] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11263.499428] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11272.981467] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11275.886412] Lustre: Permanently reactivating lustre-OST0000 [11275.971033] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11275.988162] Lustre: Skipped 2 previous similar messages [11301.855648] LustreError: 397461:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11309.358538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11309.794893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11315.872808] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11323.853827] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11331.554594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11331.560461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.577218] Lustre: Skipped 1 previous similar message [11331.585769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11336.446426] Lustre: server umount lustre-OST0000 complete [11342.821862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11342.834927] Lustre: Skipped 1 previous similar message [11346.538301] Lustre: server umount lustre-MDT0000 complete [11347.949740] LustreError: 395735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.998110] LustreError: 395735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11351.182479] LustreError: 395714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282785 with bad export cookie 2859107097081594700 [11351.191496] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11351.199545] LustreError: 395714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11351.698893] Lustre: server umount lustre-MDT0001 complete [11364.793232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11368.973234] Key type lgssc unregistered [11369.383708] LNet: 400006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11369.404903] LNetError: 400006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11369.437956] LNet: Removed LNI 192.168.203.154@tcp [11370.372514] Key type .llcrypt unregistered [11370.377169] Key type ._llcrypt unregistered [11393.003316] Key type ._llcrypt registered [11393.011688] Key type .llcrypt registered [11393.202797] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [11414.053716] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11415.482100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.594233] alg: No test for adler32 (adler32-zlib) [11416.911956] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [11417.256310] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11419.039365] Key type lgssc registered [11420.353598] Lustre: Echo OBD driver; http://www.lustre.org/ [11463.767248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11463.820472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11465.177372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11465.230793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11465.400440] Lustre: lustre-MDT0000: new disk, initializing [11465.500407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11465.518305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11470.517710] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11482.430745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.588196] Lustre: 403169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11482.692708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11482.699704] Lustre: Skipped 1 previous similar message [11482.862083] Lustre: lustre-MDT0001: new disk, initializing [11483.037783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11483.094797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11483.119519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11489.155919] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11501.594176] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11507.907355] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11515.727336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11516.028322] Lustre: lustre-OST0000: new disk, initializing [11516.033579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11516.048448] Lustre: 405140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11516.182427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11524.194673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11524.210185] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11524.277442] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11524.542929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11536.194558] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11542.443447] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11542.693334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11548.057280] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11548.332451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11555.298715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11555.302714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11555.321160] LustreError: Skipped 1 previous similar message [11555.328995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11555.336797] Lustre: Skipped 1 previous similar message [11555.367659] Lustre: Skipped 1 previous similar message [11556.776427] Lustre: server umount lustre-OST0000 complete [11565.024955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11565.035343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11565.050357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11567.008473] Lustre: server umount lustre-MDT0000 complete [11571.021350] LustreError: 403159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788283005 with bad export cookie 12455795717194524746 [11571.025028] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11571.028574] LustreError: 403159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11571.396821] Lustre: server umount lustre-MDT0001 complete [11582.992675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11585.366281] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:16:58 (1788283018) [11593.372798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11593.745478] LustreError: 407602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.763935] LustreError: 407602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11593.813316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11598.396746] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11598.828167] LustreError: 407603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.937311] LustreError: 407602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11608.055209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11608.433554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11612.941773] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11619.966461] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11625.974269] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11632.532873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11632.800662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11638.916286] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11648.152502] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11650.547821] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11653.359921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11659.746742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11659.748547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11659.757384] LustreError: Skipped 1 previous similar message [11659.758362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11659.758366] Lustre: Skipped 1 previous similar message [11659.779935] Lustre: Skipped 2 previous similar messages [11664.872154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11664.884862] Lustre: Skipped 2 previous similar messages [11666.115995] Lustre: server umount lustre-OST0000 complete [11675.118500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11675.119830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11675.148538] Lustre: Skipped 1 previous similar message [11675.156155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11675.162618] Lustre: Skipped 1 previous similar message [11685.343917] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11685.346230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11685.380609] Lustre: Skipped 3 previous similar messages [11685.613166] Lustre: server umount lustre-MDT0000 complete [11690.465316] LustreError: 407602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11690.514600] LustreError: 407602:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11690.560408] LustreError: 407584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788283125 with bad export cookie 12455795717194525628 [11690.563498] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11690.569549] LustreError: 407584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11691.008747] Lustre: server umount lustre-MDT0001 complete [11705.496115] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11709.054798] Key type lgssc unregistered [11709.516416] LNet: 411334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11709.533125] LNetError: 411334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11709.572216] LNet: Removed LNI 192.168.203.154@tcp [11710.887928] Key type .llcrypt unregistered [11710.892757] Key type ._llcrypt unregistered [11741.693414] Key type ._llcrypt registered [11741.697106] Key type .llcrypt registered [11741.865863] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11742.220515] Key type .llcrypt unregistered [11742.226404] Key type ._llcrypt unregistered [11762.498049] Key type ._llcrypt registered [11762.501050] Key type .llcrypt registered [11762.630164] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11764.230770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11764.267500] alg: No test for adler32 (adler32-zlib) [11765.390541] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [11765.661340] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11767.351213] Key type lgssc registered [11768.355378] Lustre: Echo OBD driver; http://www.lustre.org/ [11776.065520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11776.076878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.566441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11782.511105] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11787.756747] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11787.783788] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11790.855346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11791.169984] LustreError: 413552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11791.187945] LustreError: 413552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11791.189231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11796.448728] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11796.477538] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11797.892179] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11801.572115] LustreError: 413546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11801.587774] LustreError: 413546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11802.604437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11805.288294] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11807.720183] LustreError: 413548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11807.749335] LustreError: 413548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11811.854385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.165975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11816.742514] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11827.574679] Lustre: server umount lustre-OST0000 complete [11831.803282] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11831.812025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11831.833365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.802893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.807087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11832.825660] Lustre: Skipped 1 previous similar message [11837.922159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11837.937744] Lustre: Skipped 1 previous similar message [11842.098196] Lustre: server umount lustre-MDT0000 complete [11843.042305] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11843.074860] LustreError: 413551:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11844.066228] Lustre: 412863:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788283262/real 1788283262] req@00000000c114a090 x1875150844749440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788283278 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11844.115245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11845.531330] LustreError: 413534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788283280 with bad export cookie 9289265592260146943 [11845.532546] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11845.540500] LustreError: 413534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11845.836166] Lustre: server umount lustre-MDT0001 complete [11855.634889] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [11858.198248] Key type lgssc unregistered [11858.515836] LNet: 416703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11858.520711] LNetError: 416703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11858.545513] LNet: Removed LNI 192.168.203.154@tcp [11859.339845] Key type .llcrypt unregistered [11859.342934] Key type ._llcrypt unregistered [11883.984751] Key type ._llcrypt registered [11883.986688] Key type .llcrypt registered [11884.070232] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [11884.703021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11884.764659] alg: No test for adler32 (adler32-zlib) [11885.959286] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [11886.303865] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [11887.967192] Key type lgssc registered [11889.178541] Lustre: Echo OBD driver; http://www.lustre.org/ [11898.469079] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11898.486145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11914.143519] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11919.263325] LustreError: 418095:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.154@tcp: failed processing log, type 4: rc = -110 [11946.015228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11951.353814] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11959.544674] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11966.453751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.635723] LustreError: 417429:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000971eae7a x1875150970554624/t0(0) o250->MGC192.168.203.154@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11967.123948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11967.206370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11971.359433] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11972.576084] LustreError: 418121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.590700] LustreError: 418121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11977.700168] LustreError: 418120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11980.359117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11980.649284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11985.010647] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [11992.988964] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12000.713237] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12011.488890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12011.507569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.549052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.526626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.556133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12016.726433] Lustre: server umount lustre-OST0000 complete [12022.760712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12022.769916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12022.782941] Lustre: Skipped 1 previous similar message [12022.800967] Lustre: Skipped 2 previous similar messages [12027.873171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12027.895163] Lustre: Skipped 1 previous similar message [12028.214608] Lustre: server umount lustre-MDT0000 complete [12032.230619] LustreError: 418103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788283466 with bad export cookie 13585669421310262060 [12032.244162] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12032.839685] Lustre: server umount lustre-MDT0001 complete [12045.324904] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12049.024357] Key type lgssc unregistered [12049.376811] LNet: 421670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12049.387314] LNetError: 421670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12049.406690] LNet: Removed LNI 192.168.203.154@tcp [12050.394842] Key type .llcrypt unregistered [12050.396949] Key type ._llcrypt unregistered [12077.221886] Key type ._llcrypt registered [12077.227090] Key type .llcrypt registered [12077.414625] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12078.653293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12078.760335] alg: No test for adler32 (adler32-zlib) [12079.898472] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [12080.224739] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12081.967324] Key type lgssc registered [12083.435830] Lustre: Echo OBD driver; http://www.lustre.org/ [12092.497695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12092.521627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12094.081679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12099.487340] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12103.651054] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.663187] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12108.777524] LustreError: 423087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.314753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12109.831664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12115.280067] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12124.834279] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12131.686554] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12138.701583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.051672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12140.100748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12145.820389] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12154.428547] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12160.481452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12160.486246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12160.514576] Lustre: Skipped 2 previous similar messages [12160.551260] Lustre: Skipped 2 previous similar messages [12162.200296] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.54@tcp (stopping) [12165.605215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12165.622486] Lustre: Skipped 2 previous similar messages [12166.317132] Lustre: server umount lustre-MDT0000 complete [12167.251543] LustreError: 424522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.54@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12169.557804] LustreError: 423068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.54@tcp arrived at 1788283604 with bad export cookie 9820224103404909239 [12169.581181] LustreError: 423068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12170.735726] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.799360] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12172.292207] LustreError: 423067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788283606 with bad export cookie 9820224103404908343 [12172.299893] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.302535] LustreError: 423067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12175.840740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12175.845642] LustreError: 423084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.846611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12175.862341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12175.874214] LustreError: 423084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12178.725732] Lustre: server umount lustre-OST0000 complete [12186.210494] Lustre: server umount lustre-MDT0001 complete [12198.480234] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12202.507262] Key type lgssc unregistered [12202.887923] LNet: 426777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12202.893270] LNetError: 426777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12202.926734] LNet: Removed LNI 192.168.203.154@tcp [12204.288165] Key type .llcrypt unregistered [12204.293918] Key type ._llcrypt unregistered [12231.968354] Key type ._llcrypt registered [12231.970228] Key type .llcrypt registered [12232.137582] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12233.981387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12234.001552] alg: No test for adler32 (adler32-zlib) [12235.365919] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [12236.075982] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12237.895698] Key type lgssc registered [12240.701994] Lustre: Echo OBD driver; http://www.lustre.org/ [12252.569410] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12252.592153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12254.380178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12259.576695] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12263.904938] LustreError: 428197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.924948] LustreError: 428197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12269.032927] LustreError: 428198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.462107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.943482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12275.613113] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12284.050603] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12289.743511] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12297.202037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12297.547866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12298.627084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12305.361754] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12313.664884] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12320.227212] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12320.255654] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12321.427767] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12321.459522] Lustre: Skipped 1 previous similar message [12321.630180] Lustre: server umount lustre-MDT0001 complete [12325.345909] LustreError: 428197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12325.363669] LustreError: 428197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12329.445920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12329.460720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.474764] Lustre: Skipped 1 previous similar message [12329.482297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12330.465200] LustreError: 428198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12335.560845] Lustre: server umount lustre-OST0000 complete [12335.593102] LustreError: 428928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.197204] Lustre: server umount lustre-MDT0000 complete [12354.575796] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12358.188750] Key type lgssc unregistered [12358.533102] LNet: 431887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12358.540922] LNetError: 431887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12358.565890] LNet: Removed LNI 192.168.203.154@tcp [12359.677646] Key type .llcrypt unregistered [12359.679960] Key type ._llcrypt unregistered [12387.725406] Key type ._llcrypt registered [12387.734199] Key type .llcrypt registered [12387.923902] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [12405.368849] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12407.453853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12407.497029] alg: No test for adler32 (adler32-zlib) [12408.748659] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [12409.203410] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12410.911150] Key type lgssc registered [12412.219714] Lustre: Echo OBD driver; http://www.lustre.org/ [12458.515342] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12475.132054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12475.167802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12476.458458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12476.503348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12476.627470] Lustre: lustre-MDT0000: new disk, initializing [12476.738271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12476.767549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12483.226799] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12501.483137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12501.823405] Lustre: lustre-OST0000: new disk, initializing [12501.837583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12501.848094] Lustre: Skipped 1 previous similar message [12501.856898] Lustre: 436927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12502.034740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12509.867288] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12511.310512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12511.321450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12511.378633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12519.988477] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12536.755029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.894304] Lustre: 435910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12537.002486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12537.057650] Lustre: lustre-MDT0001: new disk, initializing [12537.177454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12537.221117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12537.233536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12541.990387] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12551.921414] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12552.167078] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12557.089352] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12557.305227] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12563.068236] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12567.011611] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12567.011916] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.027031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12567.043096] Lustre: Skipped 1 previous similar message [12568.163098] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.54@tcp (stopping) [12568.173667] Lustre: Skipped 1 previous similar message [12569.067394] Lustre: server umount lustre-MDT0001 complete [12577.264555] LustreError: 435916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12577.294699] LustreError: 435916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12582.396212] LustreError: 438241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12582.415384] LustreError: 438241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12587.495366] LustreError: 438241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12587.527080] LustreError: 438241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12588.006830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12588.012173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12588.035418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12592.612383] LustreError: 435921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12592.612972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12594.129245] Lustre: server umount lustre-OST0000 complete [12597.741223] LustreError: 435920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.022697] Lustre: server umount lustre-MDT0000 complete [12620.851964] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12624.279243] Key type lgssc unregistered [12624.634832] LNet: 440681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12624.646100] LNetError: 440681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12624.668707] LNet: Removed LNI 192.168.203.154@tcp [12625.604733] Key type .llcrypt unregistered [12625.610281] Key type ._llcrypt unregistered [12671.616559] Key type ._llcrypt registered [12671.618950] Key type .llcrypt registered [12671.748990] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [12672.049482] Key type .llcrypt unregistered [12672.056305] Key type ._llcrypt unregistered [12692.706382] Key type ._llcrypt registered [12692.711120] Key type .llcrypt registered [12692.877880] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [12694.257863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12694.371839] alg: No test for adler32 (adler32-zlib) [12695.806144] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [12696.201075] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [12697.967272] Key type lgssc registered [12699.427511] Lustre: Echo OBD driver; http://www.lustre.org/ [12730.055612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12730.105387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12731.361152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12731.397160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12731.476725] Lustre: lustre-MDT0000: new disk, initializing [12731.534417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12731.551950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12736.893423] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12747.898086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12748.041512] Lustre: 444063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12748.100779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12748.106334] Lustre: Skipped 1 previous similar message [12748.198517] Lustre: lustre-MDT0001: new disk, initializing [12748.324513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12748.379389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12748.397320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12753.525066] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12764.931366] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12772.455166] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12781.302344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12781.893493] Lustre: lustre-OST0000: new disk, initializing [12781.900677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12781.910557] Lustre: 446035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12782.142652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12788.903522] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12791.344583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12791.353605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12791.488124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12799.479513] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12807.142808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.151304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12807.158230] Lustre: Skipped 2 previous similar messages [12807.161663] Lustre: Skipped 1 previous similar message [12812.176356] Lustre: server umount lustre-MDT0000 complete [12815.720582] LustreError: 444056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788284250 with bad export cookie 3107890768734235686 [12815.727170] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12815.733204] LustreError: 444056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12816.021630] Lustre: server umount lustre-MDT0001 complete [12830.265426] Lustre: server umount lustre-OST0000 complete [12843.825226] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:37:56 (1788284276) [12852.374262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.901309] LustreError: 448065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12852.928291] LustreError: 448065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12853.002142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12858.336992] LustreError: 448066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12858.962602] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12863.469312] LustreError: 448065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.168371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12867.489914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12872.225746] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12879.583392] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12885.607870] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12905.840285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12906.125400] Lustre: Found index 0 for lustre-OST0000, updating log [12906.350317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12913.132356] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [12920.908071] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12923.763656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12933.097760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12933.104669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12933.118976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12933.124696] Lustre: Skipped 1 previous similar message [12934.180800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12934.190658] Lustre: Skipped 1 previous similar message [12940.261219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12940.267361] Lustre: Skipped 2 previous similar messages [12945.375270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12945.618574] Lustre: server umount lustre-OST0000 complete [12964.833802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12964.847209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12964.875508] Lustre: Skipped 1 previous similar message [12964.887135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12966.883868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12968.293431] Lustre: server umount lustre-MDT0000 complete [12970.989371] LustreError: 449882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12971.032974] LustreError: 449882:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12973.256424] LustreError: 448045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788284407 with bad export cookie 3107890768734237079 [12973.279843] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12973.728160] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [12973.744635] Lustre: server umount lustre-MDT0001 complete [12984.166807] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:40:17 (1788284417) [13613.035825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13613.637732] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.651166] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13613.700480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13619.170317] LustreError: 454479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13619.396478] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13624.305489] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13629.977774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13630.297511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13635.181948] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13643.170598] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13649.786704] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13663.967508] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13671.150029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13671.393192] Lustre: Found index 0 for lustre-OST0000, updating log [13671.743066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13679.097806] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [13690.285381] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13694.888792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13694.928582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13694.947321] Lustre: Skipped 2 previous similar messages [13708.257894] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13708.648497] Lustre: server umount lustre-OST0000 complete [13714.539477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13714.553166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13714.559252] Lustre: Skipped 1 previous similar message [13716.643342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13716.647602] Lustre: Skipped 2 previous similar messages [13719.781280] LustreError: 456214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13719.792194] LustreError: 456214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13719.858609] Lustre: server umount lustre-MDT0000 complete [13723.882979] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13723.922869] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [13724.057741] LustreError: 454459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788285158 with bad export cookie 3107890768734238164 [13724.083579] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13724.086277] LustreError: 454459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13724.520827] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp [13724.560805] Lustre: server umount lustre-MDT0001 complete [13760.974261] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [13774.673818] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13815.056772] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:54:07 (1788285247) [13833.752089] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [13836.496565] Key type lgssc unregistered [13836.733194] LNet: 461103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13836.737198] LNetError: 461103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13932.203968] LNet: Removed LNI 192.168.203.154@tcp [13932.865220] Key type .llcrypt unregistered [13932.872474] Key type ._llcrypt unregistered [13952.280479] Key type ._llcrypt registered [13952.282514] Key type .llcrypt registered [13952.464791] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [13953.568863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13953.645312] alg: No test for adler32 (adler32-zlib) [13954.776919] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [13954.973510] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [13956.687224] Key type lgssc registered [13958.382525] Lustre: Echo OBD driver; http://www.lustre.org/ [13963.593977] LNet: Added LNI 192.168.203.154@tcp100 [8/256/0/180] [13964.839948] LNet: Added LNI 192.168.203.154@tcp101 [8/256/0/180] [13965.813875] LNet: Added LNI 192.168.203.154@tcp102 [8/256/0/180] [13966.863403] LNet: Added LNI 192.168.203.154@tcp103 [8/256/0/180] [13967.877492] LNet: Added LNI 192.168.203.154@tcp104 [8/256/0/180] [13969.129171] LNet: Added LNI 192.168.203.154@tcp105 [8/256/0/180] [13970.195276] LNet: Added LNI 192.168.203.154@tcp106 [8/256/0/180] [13971.245933] LNet: Added LNI 192.168.203.154@tcp107 [8/256/0/180] [13972.227035] LNet: Added LNI 192.168.203.154@tcp108 [8/256/0/180] [13973.191892] LNet: Added LNI 192.168.203.154@tcp109 [8/256/0/180] [13974.196276] LNet: Added LNI 192.168.203.154@tcp110 [8/256/0/180] [13975.217993] LNet: Added LNI 192.168.203.154@tcp111 [8/256/0/180] [13976.368054] LNet: Added LNI 192.168.203.154@tcp112 [8/256/0/180] [13977.339241] LNet: Added LNI 192.168.203.154@tcp113 [8/256/0/180] [13978.207819] LNet: Added LNI 192.168.203.154@tcp114 [8/256/0/180] [13979.091726] LNet: Added LNI 192.168.203.154@tcp115 [8/256/0/180] [13980.106228] LNet: Added LNI 192.168.203.154@tcp116 [8/256/0/180] [13981.249983] LNet: Added LNI 192.168.203.154@tcp117 [8/256/0/180] [13982.295047] LNet: Added LNI 192.168.203.154@tcp118 [8/256/0/180] [13983.253515] LNet: Added LNI 192.168.203.154@tcp119 [8/256/0/180] [13984.244467] LNet: Added LNI 192.168.203.154@tcp120 [8/256/0/180] [13985.109899] LNet: Added LNI 192.168.203.154@tcp121 [8/256/0/180] [13986.043784] LNet: Added LNI 192.168.203.154@tcp122 [8/256/0/180] [13986.997771] LNet: Added LNI 192.168.203.154@tcp123 [8/256/0/180] [13988.002358] LNet: Added LNI 192.168.203.154@tcp124 [8/256/0/180] [13988.897733] LNet: Added LNI 192.168.203.154@tcp125 [8/256/0/180] [13989.959993] LNet: Added LNI 192.168.203.154@tcp126 [8/256/0/180] [13990.842943] LNet: Added LNI 192.168.203.154@tcp127 [8/256/0/180] [13991.838807] LNet: Added LNI 192.168.203.154@tcp128 [8/256/0/180] [13992.796413] LNet: Added LNI 192.168.203.154@tcp129 [8/256/0/180] [13993.777898] LNet: Added LNI 192.168.203.154@tcp130 [8/256/0/180] [13994.747727] LNet: Added LNI 192.168.203.154@tcp131 [8/256/0/180] [13995.708823] LNet: Added LNI 192.168.203.154@tcp132 [8/256/0/180] [13996.679704] LNet: Added LNI 192.168.203.154@tcp133 [8/256/0/180] [13997.546354] LNet: Added LNI 192.168.203.154@tcp134 [8/256/0/180] [13998.590372] LNet: Added LNI 192.168.203.154@tcp135 [8/256/0/180] [14005.783935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14005.803903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14006.939494] Lustre: 464151:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [14007.018332] LNetError: 461610:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14007.029579] LNetError: Protocol error connecting to 192.168.203.154@tcp121 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14007.055495] LNetError: Skipped 5 previous similar messages [14007.170290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14007.200530] LNet: 6 local NIs in recovery (showing 6): 192.168.203.154@tcp124, 192.168.203.154@tcp125, 192.168.203.154@tcp127, 192.168.203.154@tcp128, 192.168.203.154@tcp121, 192.168.203.154@tcp129 [14007.245382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14007.330749] Lustre: lustre-MDT0000: new disk, initializing [14007.421027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14007.441468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14008.096231] LNetError: 461609:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14008.113077] LNetError: 461609:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [14008.126853] LNetError: Protocol error connecting to 192.168.203.154@tcp120 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14008.146426] LNetError: Skipped 8 previous similar messages [14010.273942] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14010.296032] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14010.307823] LNetError: Protocol error connecting to 192.168.203.154@tcp124 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14010.321061] LNetError: Skipped 8 previous similar messages [14012.649050] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14014.372200] LNetError: 464175:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14014.386358] LNetError: 464175:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14014.401073] LNetError: Protocol error connecting to 192.168.203.154@tcp124 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14022.564575] LNetError: 464184:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14022.574203] LNetError: Protocol error connecting to 192.168.203.154@tcp124 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14023.659512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14023.869946] Lustre: 464168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14023.957473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14023.969829] Lustre: Skipped 1 previous similar message [14024.070176] Lustre: lustre-MDT0001: new disk, initializing [14024.099604] Lustre: 464969:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [14024.109392] Lustre: 464969:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14024.205419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14024.245041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14024.259916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14029.248372] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14031.403484] Lustre: 465332:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [14031.416983] Lustre: 465332:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14037.921090] LNetError: 464171:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14037.937749] LNetError: Protocol error connecting to 192.168.203.154@tcp109 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14039.711676] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14045.731292] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14053.044178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14053.497939] Lustre: lustre-OST0000: new disk, initializing [14053.500994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14053.508307] Lustre: 466140:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14053.627462] Lustre: 466145:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [14053.632871] Lustre: 466145:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14053.678148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14054.305905] LNetError: 464184:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14054.314738] LNetError: 464184:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 18 previous similar messages [14054.321411] LNetError: Protocol error connecting to 192.168.203.154@tcp123 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14054.331089] LNetError: Skipped 18 previous similar messages [14060.098941] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14060.560553] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.154@tcp: can't add NID 192.168.203.154@tcp130: rc = -75 [14060.570635] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14060.645917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14060.667703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14060.874712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14066.599452] LNet: 1 local NIs in recovery (showing 1): 192.168.203.154@tcp124 [14071.399992] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14081.504295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14081.506946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14081.519749] LustreError: Skipped 1 previous similar message [14081.522767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14081.549442] Lustre: Skipped 1 previous similar message [14086.719101] Lustre: server umount lustre-OST0000 complete [14088.096404] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.154 [14088.109165] LNetError: 464178:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 22 previous similar messages [14088.128275] LNetError: Protocol error connecting to 192.168.203.154@tcp122 at host 192.168.203.154:988: is it running a compatible version of Lustre? [14088.140719] LNetError: Skipped 22 previous similar messages [14091.745317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14091.749300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14091.778778] Lustre: Skipped 1 previous similar message [14091.788518] Lustre: Skipped 1 previous similar message [14096.866215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14096.879943] Lustre: Skipped 2 previous similar messages [14097.334537] Lustre: server umount lustre-MDT0000 complete [14101.654416] LustreError: 464159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788285536 with bad export cookie 7605275697782874958 [14101.656314] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14101.669399] LustreError: 464159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14101.932191] Lustre: server umount lustre-MDT0001 complete [14114.443070] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14117.289988] Key type lgssc unregistered [14117.648980] LNet: 468059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14117.663761] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14117.670423] LNet: Removed LNI 192.168.203.154@tcp [14118.689716] LNet: Removed LNI 192.168.203.154@tcp100 [14118.697226] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14118.716837] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14118.735308] LNet: Removed LNI 192.168.203.154@tcp101 [14119.775309] LNet: Removed LNI 192.168.203.154@tcp102 [14119.778224] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14119.782862] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14119.787579] LNet: Removed LNI 192.168.203.154@tcp103 [14120.799774] LNet: Removed LNI 192.168.203.154@tcp104 [14121.823162] LNet: Removed LNI 192.168.203.154@tcp105 [14121.826138] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14121.847599] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14121.867570] LNet: Removed LNI 192.168.203.154@tcp106 [14122.911181] LNet: Removed LNI 192.168.203.154@tcp107 [14123.935214] LNet: Removed LNI 192.168.203.154@tcp108 [14124.959907] LNet: Removed LNI 192.168.203.154@tcp109 [14125.983387] LNet: Removed LNI 192.168.203.154@tcp110 [14125.989429] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14126.013695] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14126.020553] LNet: Removed LNI 192.168.203.154@tcp111 [14127.071181] LNet: Removed LNI 192.168.203.154@tcp112 [14128.095635] LNet: Removed LNI 192.168.203.154@tcp113 [14129.119219] LNet: Removed LNI 192.168.203.154@tcp114 [14130.146354] LNet: Removed LNI 192.168.203.154@tcp115 [14131.168234] LNet: Removed LNI 192.168.203.154@tcp116 [14132.255262] LNet: Removed LNI 192.168.203.154@tcp117 [14133.343217] LNet: Removed LNI 192.168.203.154@tcp118 [14134.369966] LNet: Removed LNI 192.168.203.154@tcp119 [14134.381549] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14134.393965] LNetError: 468059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14134.404492] LNet: Removed LNI 192.168.203.154@tcp120 [14135.456670] LNet: Removed LNI 192.168.203.154@tcp121 [14136.479159] LNet: Removed LNI 192.168.203.154@tcp122 [14137.506488] LNet: Removed LNI 192.168.203.154@tcp123 [14137.508669] LNet: Removed LNI 192.168.203.154@tcp124 [14138.528097] LNet: Removed LNI 192.168.203.154@tcp125 [14139.551283] LNet: Removed LNI 192.168.203.154@tcp126 [14140.575241] LNet: Removed LNI 192.168.203.154@tcp127 [14141.601285] LNet: Removed LNI 192.168.203.154@tcp128 [14142.623205] LNet: Removed LNI 192.168.203.154@tcp129 [14142.634473] LNet: Removed LNI 192.168.203.154@tcp130 [14142.635918] LNet: Removed LNI 192.168.203.154@tcp131 [14142.642304] LNet: Removed LNI 192.168.203.154@tcp132 [14142.647881] LNet: Removed LNI 192.168.203.154@tcp133 [14142.651364] LNet: Removed LNI 192.168.203.154@tcp134 [14142.662341] LNet: Removed LNI 192.168.203.154@tcp135 [14143.524913] Key type .llcrypt unregistered [14143.526534] Key type ._llcrypt unregistered [14172.777199] Key type ._llcrypt registered [14172.779732] Key type .llcrypt registered [14172.933916] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14190.071745] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14191.864199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14192.138032] alg: No test for adler32 (adler32-zlib) [14193.264452] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [14193.486653] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14195.136091] Key type lgssc registered [14196.653162] Lustre: Echo OBD driver; http://www.lustre.org/ [14237.872815] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:01:10 (1788285670) [14257.496307] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14260.486138] Key type lgssc unregistered [14260.756481] LNet: 472165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14260.764686] LNetError: 472165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14260.776765] LNet: Removed LNI 192.168.203.154@tcp [14261.622336] Key type .llcrypt unregistered [14261.624157] Key type ._llcrypt unregistered [14282.111277] Key type ._llcrypt registered [14282.114403] Key type .llcrypt registered [14282.226417] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14283.244987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14283.293888] alg: No test for adler32 (adler32-zlib) [14284.450875] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [14284.722342] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14286.466716] Key type lgssc registered [14288.006208] Lustre: Echo OBD driver; http://www.lustre.org/ [14298.984668] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14299.050790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14300.344211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14300.375142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14300.475938] Lustre: lustre-MDT0000: new disk, initializing [14300.560267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14300.579122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14305.603160] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14317.594657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14317.815570] Lustre: 473381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14317.903763] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14317.908367] Lustre: Skipped 1 previous similar message [14317.992400] Lustre: lustre-MDT0001: new disk, initializing [14318.081765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14318.122870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14318.143522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14323.549885] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14333.690310] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14339.453118] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14348.003701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14348.285966] Lustre: lustre-OST0000: new disk, initializing [14348.289962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14348.307865] Lustre: 475350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14348.465600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14356.170260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14356.201749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14356.207348] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14356.350535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14369.102438] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14375.317823] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing lnet_if_list [14381.816844] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14381.870112] LNet: Added LNI 192.168.203.154@tcp10 [8/256/0/180] [14388.540706] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14388.578240] LNet: Added LNI 192.168.203.154@tcp11 [8/256/0/180] [14393.073701] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14393.099328] LNet: Added LNI 192.168.203.154@tcp12 [8/256/0/180] [14398.086781] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14398.126052] LNet: Added LNI 192.168.203.154@tcp13 [8/256/0/180] [14403.333783] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14403.378827] LNet: Added LNI 192.168.203.154@tcp14 [8/256/0/180] [14407.979791] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14408.029364] LNet: Added LNI 192.168.203.154@tcp15 [8/256/0/180] [14413.613996] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14413.654663] LNet: Added LNI 192.168.203.154@tcp16 [8/256/0/180] [14418.824408] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14418.853882] LNet: Added LNI 192.168.203.154@tcp17 [8/256/0/180] [14424.656344] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14424.685902] LNet: Added LNI 192.168.203.154@tcp18 [8/256/0/180] [14430.583491] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14430.640653] LNet: Added LNI 192.168.203.154@tcp19 [8/256/0/180] [14436.891311] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14436.919503] LNet: Added LNI 192.168.203.154@tcp20 [8/256/0/180] [14442.931621] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14442.958238] LNet: Added LNI 192.168.203.154@tcp21 [8/256/0/180] [14448.800694] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14448.860502] LNet: Added LNI 192.168.203.154@tcp22 [8/256/0/180] [14456.930460] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14456.974280] LNet: Added LNI 192.168.203.154@tcp23 [8/256/0/180] [14465.329669] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14465.425767] LNet: Added LNI 192.168.203.154@tcp24 [8/256/0/180] [14476.069581] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14476.107983] LNet: Added LNI 192.168.203.154@tcp25 [8/256/0/180] [14482.381133] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14482.400215] LNet: Added LNI 192.168.203.154@tcp26 [8/256/0/180] [14489.249497] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14489.319978] LNet: Added LNI 192.168.203.154@tcp27 [8/256/0/180] [14495.050541] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14495.102426] LNet: Added LNI 192.168.203.154@tcp28 [8/256/0/180] [14502.881508] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14502.931056] LNet: Added LNI 192.168.203.154@tcp29 [8/256/0/180] [14509.800348] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14509.831246] LNet: Added LNI 192.168.203.154@tcp30 [8/256/0/180] [14516.388932] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14516.418452] LNet: Added LNI 192.168.203.154@tcp31 [8/256/0/180] [14523.574380] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14523.607419] LNet: Added LNI 192.168.203.154@tcp32 [8/256/0/180] [14530.490538] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14530.516800] LNet: Added LNI 192.168.203.154@tcp33 [8/256/0/180] [14538.192970] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14538.266656] LNet: Added LNI 192.168.203.154@tcp34 [8/256/0/180] [14546.897404] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14546.935161] LNet: Added LNI 192.168.203.154@tcp35 [8/256/0/180] [14554.760480] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14554.818377] LNet: Added LNI 192.168.203.154@tcp36 [8/256/0/180] [14563.604856] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14563.646744] LNet: Added LNI 192.168.203.154@tcp37 [8/256/0/180] [14572.275775] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14572.315900] LNet: Added LNI 192.168.203.154@tcp38 [8/256/0/180] [14580.094575] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14580.140202] LNet: Added LNI 192.168.203.154@tcp39 [8/256/0/180] [14588.521134] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14588.574417] LNet: Added LNI 192.168.203.154@tcp40 [8/256/0/180] [14597.126728] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14597.171825] LNet: Added LNI 192.168.203.154@tcp41 [8/256/0/180] [14606.011285] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14606.041583] LNet: Added LNI 192.168.203.154@tcp42 [8/256/0/180] [14615.617361] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14615.647464] LNet: Added LNI 192.168.203.154@tcp43 [8/256/0/180] [14625.011486] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14625.055727] LNet: Added LNI 192.168.203.154@tcp44 [8/256/0/180] [14634.716522] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14634.764673] LNet: Added LNI 192.168.203.154@tcp45 [8/256/0/180] [14644.170561] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14644.238833] LNet: Added LNI 192.168.203.154@tcp46 [8/256/0/180] [14653.178442] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14653.232903] LNet: Added LNI 192.168.203.154@tcp47 [8/256/0/180] [14662.720282] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14662.773776] LNet: Added LNI 192.168.203.154@tcp48 [8/256/0/180] [14673.672600] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14673.706528] LNet: Added LNI 192.168.203.154@tcp49 [8/256/0/180] [14684.213752] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14684.248739] LNet: Added LNI 192.168.203.154@tcp50 [8/256/0/180] [14704.617793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14704.620767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14704.635531] Lustre: Skipped 2 previous similar messages [14704.646694] Lustre: Skipped 1 previous similar message [14709.736451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14709.755543] Lustre: Skipped 3 previous similar messages [14709.967719] Lustre: server umount lustre-MDT0000 complete [14714.445807] LustreError: 473372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788286148 with bad export cookie 9488695603081563395 [14714.455484] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14714.468153] LustreError: 473372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14714.820704] Lustre: server umount lustre-MDT0001 complete [14731.103263] Lustre: 472650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788286149/real 1788286149] req@000000003a362fcb x1875153486273408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788286165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14731.156155] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14735.043213] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14736.289179] Lustre: 472650:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788286154/real 1788286154] req@00000000a88c5636 x1875153486273536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788286170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14740.453878] Lustre: 472653:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788286158/real 1788286158] req@00000000539c4320 x1875153486273792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788286174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14743.519468] Lustre: 472652:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788286161/real 1788286161] req@0000000066fedb13 x1875153486273920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788286177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14745.401252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14758.946712] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83ae9e60b37cd76e [14758.958684] Lustre: MGC192.168.203.154@tcp: Connection restored to 0@lo (at 0@lo) [14759.393889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14760.430316] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14764.442599] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14766.568093] LustreError: 475345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14766.590820] LustreError: 475345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14771.689101] LustreError: 475344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14773.999668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14774.490394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14776.574867] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14779.564190] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14787.784480] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14793.597973] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14815.712833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14815.716544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14815.722384] LustreError: Skipped 1 previous similar message [14815.725424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14815.756421] Lustre: Skipped 1 previous similar message [14817.765250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14817.784479] Lustre: Skipped 2 previous similar messages [14820.285470] Lustre: server umount lustre-OST0000 complete [14825.954501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14825.971731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14825.995449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14830.757971] Lustre: server umount lustre-MDT0000 complete [14831.011256] LustreError: 499607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14831.032726] LustreError: 499607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14834.521708] LustreError: 473371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788286269 with bad export cookie 9488695603081566062 [14834.523109] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14834.536463] LustreError: 473371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14834.964222] Lustre: server umount lustre-MDT0001 complete [14846.347330] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [14849.388936] Key type lgssc unregistered [14849.749606] LNet: 502408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14849.759924] LNetError: 502408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14850.783305] LNet: Removed LNI 192.168.203.154@tcp [14850.786438] LNetError: 502408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14850.796678] LNet: Removed LNI 192.168.203.154@tcp10 [14850.802856] LNet: Removed LNI 192.168.203.154@tcp11 [14850.808405] LNet: Removed LNI 192.168.203.154@tcp12 [14850.814591] LNet: Removed LNI 192.168.203.154@tcp13 [14850.821326] LNet: Removed LNI 192.168.203.154@tcp14 [14850.831236] LNet: Removed LNI 192.168.203.154@tcp15 [14850.839480] LNet: Removed LNI 192.168.203.154@tcp16 [14850.847501] LNet: Removed LNI 192.168.203.154@tcp17 [14850.855287] LNet: Removed LNI 192.168.203.154@tcp18 [14850.857684] LNet: Removed LNI 192.168.203.154@tcp19 [14850.864522] LNet: Removed LNI 192.168.203.154@tcp20 [14850.866807] LNet: Removed LNI 192.168.203.154@tcp21 [14850.872380] LNet: Removed LNI 192.168.203.154@tcp22 [14850.876125] LNet: Removed LNI 192.168.203.154@tcp23 [14850.880526] LNet: Removed LNI 192.168.203.154@tcp24 [14850.883571] LNet: Removed LNI 192.168.203.154@tcp25 [14850.887343] LNet: Removed LNI 192.168.203.154@tcp26 [14850.890707] LNet: Removed LNI 192.168.203.154@tcp27 [14850.893419] LNet: Removed LNI 192.168.203.154@tcp28 [14850.896721] LNet: Removed LNI 192.168.203.154@tcp29 [14850.902427] LNet: Removed LNI 192.168.203.154@tcp30 [14850.905486] LNet: Removed LNI 192.168.203.154@tcp31 [14850.908872] LNet: Removed LNI 192.168.203.154@tcp32 [14850.911466] LNet: Removed LNI 192.168.203.154@tcp33 [14850.914263] LNet: Removed LNI 192.168.203.154@tcp34 [14850.917348] LNet: Removed LNI 192.168.203.154@tcp35 [14850.921230] LNet: Removed LNI 192.168.203.154@tcp36 [14850.923890] LNet: Removed LNI 192.168.203.154@tcp37 [14850.927092] LNet: Removed LNI 192.168.203.154@tcp38 [14850.930566] LNet: Removed LNI 192.168.203.154@tcp39 [14850.933708] LNet: Removed LNI 192.168.203.154@tcp40 [14850.936638] LNet: Removed LNI 192.168.203.154@tcp41 [14850.939654] LNet: Removed LNI 192.168.203.154@tcp42 [14850.943244] LNet: Removed LNI 192.168.203.154@tcp43 [14850.949120] LNet: Removed LNI 192.168.203.154@tcp44 [14850.952225] LNet: Removed LNI 192.168.203.154@tcp45 [14850.957215] LNet: Removed LNI 192.168.203.154@tcp46 [14850.960726] LNet: Removed LNI 192.168.203.154@tcp47 [14850.963667] LNet: Removed LNI 192.168.203.154@tcp48 [14850.967335] LNet: Removed LNI 192.168.203.154@tcp49 [14850.981350] LNet: Removed LNI 192.168.203.154@tcp50 [14851.609573] Key type .llcrypt unregistered [14851.612258] Key type ._llcrypt unregistered [14873.270680] Key type ._llcrypt registered [14873.273870] Key type .llcrypt registered [14873.461969] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [14891.827630] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [14892.976169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14893.127568] alg: No test for adler32 (adler32-zlib) [14894.237907] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [14894.518553] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [14896.199239] Key type lgssc registered [14897.590809] Lustre: Echo OBD driver; http://www.lustre.org/ [14934.263169] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14934.293345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14935.688030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14935.740932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14935.858864] Lustre: lustre-MDT0000: new disk, initializing [14935.954510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14935.979657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14941.134675] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14952.441045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14952.561038] Lustre: 505608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14952.630519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14952.637195] Lustre: Skipped 1 previous similar message [14952.715686] Lustre: lustre-MDT0001: new disk, initializing [14952.836175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14952.906364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14952.923381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14957.187592] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14967.366825] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14974.147556] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14982.585533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14982.771174] Lustre: *** cfs_fail_loc=90f, val=0*** [14982.900848] Lustre: lustre-OST0000: new disk, initializing [14982.909538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14982.929602] Lustre: 507621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14983.066743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14989.491848] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [14991.495350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14991.510390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14991.707253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15002.455441] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15017.441766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15017.459938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15017.492212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15017.505711] Lustre: Skipped 1 previous similar message [15018.766267] Lustre: server umount lustre-OST0000 complete [15024.618812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15024.633796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15024.647839] Lustre: Skipped 1 previous similar message [15024.657418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15027.684961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15027.686967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15027.720577] Lustre: Skipped 1 previous similar message [15032.800475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15032.818909] Lustre: Skipped 1 previous similar message [15037.407184] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15037.679501] Lustre: server umount lustre-MDT0000 complete [15042.435485] LustreError: 505598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788286476 with bad export cookie 2577411866975597911 [15042.456258] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15042.792382] Lustre: server umount lustre-MDT0001 complete [15054.090967] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [15057.365780] Key type lgssc unregistered [15057.826532] LNet: 509533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15057.837839] LNetError: 509533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15057.852590] LNet: Removed LNI 192.168.203.154@tcp [15058.703815] Key type .llcrypt unregistered [15058.716753] Key type ._llcrypt unregistered [15080.726276] Key type ._llcrypt registered [15080.728841] Key type .llcrypt registered [15080.844518] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [15082.010839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15082.155319] alg: No test for adler32 (adler32-zlib) [15083.463429] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [15083.935244] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [15085.591538] Key type lgssc registered [15087.234052] Lustre: Echo OBD driver; http://www.lustre.org/ [15096.864158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15096.881684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15098.188451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15102.384281] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15107.569020] LustreError: 510804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15107.595859] LustreError: 510804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15111.475815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15112.008684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15116.637368] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15125.150077] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15131.945643] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15139.312890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15139.719866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15146.264183] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15154.474414] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15166.786775] loop0: detected capacity change from 0 to 204800000 [15166.984290] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15166.993552] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15167.005246] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15173.465273] loop0: detected capacity change from 0 to 204800000 [15173.673757] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15173.683968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15173.709258] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15180.093821] loop0: detected capacity change from 0 to 204800000 [15186.551347] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15186.770358] Lustre: fs15246-OST0000: new disk, initializing [15186.781547] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15186.793196] Lustre: 514399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15186.882180] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15193.974875] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15206.073957] loop1: detected capacity change from 0 to 204800000 [15213.426263] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15213.600112] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15213.710405] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15213.842344] Lustre: fs15246-MDT0000: new disk, initializing [15214.064718] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15214.110992] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15214.331425] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15215.395666] Lustre: 515524:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000007b18c40 x1875154323949824/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788286665 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15218.690677] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15224.481855] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15224.490776] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15224.517412] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15241.715053] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15241.740664] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15246.832512] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15247.064856] Lustre: server umount fs15246-MDT0000 complete [15251.969672] LustreError: 510801:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15257.063298] LustreError: 510801:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15257.355257] Lustre: server umount fs15246-OST0000 complete [15269.857043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15269.869922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15269.886482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15270.884430] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15270.900355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15272.432177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15272.439726] Lustre: Skipped 2 previous similar messages [15277.537691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15277.547276] Lustre: Skipped 1 previous similar message [15284.191255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15284.437572] Lustre: server umount lustre-OST0000 complete [15291.361325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15291.375475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15291.394714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15291.408161] Lustre: Skipped 2 previous similar messages [15294.766724] Lustre: server umount lustre-MDT0000 complete [15298.016294] LustreError: 514437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15298.041429] LustreError: 514437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15299.209077] LustreError: 510784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788286733 with bad export cookie 8207939535745561863 [15299.222968] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15299.226113] LustreError: 510784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15299.708322] Lustre: server umount lustre-MDT0001 complete [15312.832697] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [15316.136673] Key type lgssc unregistered [15316.404156] LNet: 518625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15316.420428] LNetError: 518625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15316.442307] LNet: Removed LNI 192.168.203.154@tcp [15317.370360] Key type .llcrypt unregistered [15317.373052] Key type ._llcrypt unregistered [15335.737505] Key type ._llcrypt registered [15335.741539] Key type .llcrypt registered [15335.896616] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [15337.298543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15337.325402] alg: No test for adler32 (adler32-zlib) [15338.479160] Lustre: Lustre: Build Version: 2.17.57_64_g65721f0 [15338.749704] LNet: Added LNI 192.168.203.154@tcp [8/256/0/180] [15340.463172] Key type lgssc registered [15341.844345] Lustre: Echo OBD driver; http://www.lustre.org/ [15382.212048] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [15393.177054] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [15434.264649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15434.303613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15435.603244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15435.655755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15435.744392] Lustre: lustre-MDT0000: new disk, initializing [15435.822771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15435.846481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15440.761954] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15451.637084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15451.795858] Lustre: 523108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15451.875875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15451.879728] Lustre: Skipped 1 previous similar message [15451.981926] Lustre: lustre-MDT0001: new disk, initializing [15452.105937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15452.188523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15452.207498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15456.580382] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15466.481865] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15472.585685] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15479.817882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15480.099418] Lustre: lustre-OST0000: new disk, initializing [15480.107525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15480.124464] Lustre: 525080:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15480.239815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15486.946939] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_default_debug -1 all [15490.123037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15490.137280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15490.301609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15496.390642] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15501.387722] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15501.629369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15506.220975] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15506.417336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15511.010994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15511.027626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15511.037969] Lustre: Skipped 1 previous similar message [15511.039177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15514.772717] Lustre: server umount lustre-OST0000 complete [15521.250584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15521.272422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15521.286342] Lustre: Skipped 2 previous similar messages [15525.294596] Lustre: server umount lustre-MDT0000 complete [15529.630820] LustreError: 523101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788286964 with bad export cookie 7725861561694361826 [15529.639060] LustreError: MGC192.168.203.154@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15529.648193] LustreError: 523101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15529.965347] Lustre: server umount lustre-MDT0001 complete [15558.562259] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing set_hostid [15574.390181] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing load_modules_local [15610.752219] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15377 sec ========= 14:24:03 (1788287043) [15612.562362] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:24:05 (1788287045) === [15616.156080] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:24:09 (1788287049) === [15659.613844] Lustre: DEBUG MARKER: oleg354-server.virtnet: executing unload_modules_local [15662.654461] Key type lgssc unregistered [15662.984052] LNet: 531747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15662.992155] LNetError: 531747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15663.003029] LNet: Removed LNI 192.168.203.154@tcp [15663.942752] Key type .llcrypt unregistered [15663.947178] Key type ._llcrypt unregistered