[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 550520092 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003369] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.007971] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010182] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.012060] SELinux: Initializing. [ 0.013134] *** VALIDATE selinux *** [ 0.022031] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027074] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031162] *** VALIDATE tmpfs *** [ 0.033015] *** VALIDATE proc *** [ 0.034320] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.036321] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038090] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044816] debug: unmapping init [mem 0xffffffff9f859000-0xffffffff9f860fff] [ 0.046214] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047840] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048038] ... version: 2 [ 0.049020] ... bit width: 48 [ 0.050019] ... generic registers: 4 [ 0.051022] ... value mask: 0000ffffffffffff [ 0.052023] ... max period: 00007fffffffffff [ 0.053023] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.055381] rcu: Hierarchical SRCU implementation. [ 0.057781] smp: Bringing up secondary CPUs ... [ 0.058746] x86: Booting SMP configuration: [ 0.059057] .... node #0, CPUs: #1 #2 #3 [ 0.063029] smp: Brought up 1 node, 4 CPUs [ 0.065023] smpboot: Max logical packages: 1 [ 0.066026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225773] node 0 deferred pages initialised in 156ms [ 0.230081] devtmpfs: initialized [ 0.231461] x86/mm: Memory block size: 128MB [ 0.235321] gcov: version magic: 0x41383552 [ 0.238098] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.240160] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243441] pinctrl core: initialized pinctrl subsystem [ 0.245218] [ 0.245641] ************************************************************* [ 0.247015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249010] ** ** [ 0.250008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251009] ** ** [ 0.253012] ** This means that this kernel is built to expose internal ** [ 0.254012] ** IOMMU data structures, which may compromise security on ** [ 0.256014] ** your system. ** [ 0.258014] ** ** [ 0.259007] ** If you see this message and you are not debugging the ** [ 0.261011] ** kernel, report this immediately to your vendor! ** [ 0.264015] ** ** [ 0.265017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267010] ************************************************************* [ 0.268644] NET: Registered protocol family 16 [ 0.272386] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.274100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.281057] cpuidle: using governor menu [ 0.282735] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.283620] PCI: Using configuration type 1 for base access [ 0.284169] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.292174] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.295025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.297139] cryptd: max_cpu_qlen set to 1000 [ 0.300292] ACPI: Added _OSI(Module Device) [ 0.302020] ACPI: Added _OSI(Processor Device) [ 0.304016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.305016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.310000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.317102] ACPI: Interpreter enabled [ 0.318073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320014] ACPI: Using IOAPIC for interrupt routing [ 0.322148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.325493] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.336982] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.339057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.341025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.344085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.348380] acpiphp: Slot [2] registered [ 0.350105] acpiphp: Slot [5] registered [ 0.351096] acpiphp: Slot [6] registered [ 0.352086] acpiphp: Slot [7] registered [ 0.353090] acpiphp: Slot [8] registered [ 0.354109] acpiphp: Slot [9] registered [ 0.355166] acpiphp: Slot [10] registered [ 0.356126] acpiphp: Slot [3] registered [ 0.358127] acpiphp: Slot [4] registered [ 0.359098] acpiphp: Slot [11] registered [ 0.360055] acpiphp: Slot [12] registered [ 0.361095] acpiphp: Slot [13] registered [ 0.361906] acpiphp: Slot [14] registered [ 0.363056] acpiphp: Slot [15] registered [ 0.363921] acpiphp: Slot [16] registered [ 0.365051] acpiphp: Slot [17] registered [ 0.365842] acpiphp: Slot [18] registered [ 0.366099] acpiphp: Slot [19] registered [ 0.367079] acpiphp: Slot [20] registered [ 0.368000] acpiphp: Slot [21] registered [ 0.369248] acpiphp: Slot [22] registered [ 0.370072] acpiphp: Slot [23] registered [ 0.370927] acpiphp: Slot [24] registered [ 0.372089] acpiphp: Slot [25] registered [ 0.373100] acpiphp: Slot [26] registered [ 0.373987] acpiphp: Slot [27] registered [ 0.375091] acpiphp: Slot [28] registered [ 0.375995] acpiphp: Slot [29] registered [ 0.377065] acpiphp: Slot [30] registered [ 0.379118] acpiphp: Slot [31] registered [ 0.380073] PCI host bridge to bus 0000:00 [ 0.381024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.383039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.385024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.387026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.389023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.391029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.392234] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.394857] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.397044] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.405727] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.410855] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.413021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.415019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.417019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.419637] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422092] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.424037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.427050] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.452027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.458029] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461887] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.468022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.473018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.510285] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.515019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.520021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.536020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.542529] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.563017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.570893] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.578020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.582018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.607016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.614051] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.620021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.624022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.634024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.641743] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.648019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.653016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.665018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.674251] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.676301] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.679257] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.681441] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.687105] iommu: Default domain type: Passthrough [ 0.689516] SCSI subsystem initialized [ 0.692168] ACPI: bus type USB registered [ 0.694122] usbcore: registered new interface driver usbfs [ 0.697099] usbcore: registered new interface driver hub [ 0.699121] usbcore: registered new device driver usb [ 0.702237] pps_core: LinuxPPS API ver. 1 registered [ 0.705015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.709054] PTP clock support registered [ 0.711125] EDAC MC: Ver: 3.0.0 [ 0.713027] PCI: Using ACPI for IRQ routing [ 0.714962] NetLabel: Initializing [ 0.715018] NetLabel: domain hash size = 128 [ 0.716017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.717100] NetLabel: unlabeled traffic allowed by default [ 0.719147] vgaarb: loaded [ 0.720319] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.722018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.727377] clocksource: Switched to clocksource kvm-clock [ 0.827880] VFS: Disk quotas dquot_6.6.0 [ 0.830137] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.833903] *** VALIDATE ramfs *** [ 0.835716] *** VALIDATE hugetlbfs *** [ 0.838684] pnp: PnP ACPI init [ 0.841792] pnp: PnP ACPI: found 6 devices [ 0.860720] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.864439] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.866853] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869286] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.872066] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.874807] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.878099] NET: Registered protocol family 2 [ 0.880787] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.886352] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890254] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.895856] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.899736] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902633] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.906198] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909596] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913274] NET: Registered protocol family 1 [ 0.915908] RPC: Registered named UNIX socket transport module. [ 0.918524] RPC: Registered udp transport module. [ 0.920487] RPC: Registered tcp transport module. [ 0.922360] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924823] NET: Registered protocol family 44 [ 0.926626] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.929043] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.931439] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.933886] PCI: CLS 0 bytes, default 64 [ 0.935789] Unpacking initramfs... [ 2.436613] debug: unmapping init [mem 0xffff9dc7bcc54000-0xffff9dc7bffbffff] [ 2.442057] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.447692] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.454441] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.535531] Initialise system trusted keyrings [ 3.537393] Key type blacklist registered [ 3.539452] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.549492] zbud: loaded [ 3.552684] *** VALIDATE nfs *** [ 3.554083] *** VALIDATE nfs4 *** [ 3.555780] pstore: using deflate compression [ 3.559713] Platform Keyring initialized [ 3.690614] NET: Registered protocol family 38 [ 3.692779] Key type asymmetric registered [ 3.697523] Asymmetric key parser 'x509' registered [ 3.702784] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.707727] io scheduler mq-deadline registered [ 3.710836] io scheduler kyber registered [ 3.713702] io scheduler bfq registered [ 3.716572] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.721197] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.727533] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.732599] ACPI: Power Button [PWRF] [ 3.741437] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.752559] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.772676] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.784375] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.808665] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.840235] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.872028] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.880826] Non-volatile memory driver v1.3 [ 3.883781] Linux agpgart interface v0.103 [ 3.938357] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.942536] vda: detected capacity change from 0 to 76677120 [ 3.965032] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.969703] vdb: detected capacity change from 0 to 1073741824 [ 3.995806] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.000633] vdc: detected capacity change from 0 to 2621440000 [ 4.028457] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.032472] vdd: detected capacity change from 0 to 2621440000 [ 4.059550] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.064507] vde: detected capacity change from 0 to 4294967296 [ 4.095969] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.100583] vdf: detected capacity change from 0 to 4294967296 [ 4.112747] libphy: Fixed MDIO Bus: probed [ 4.125770] usbcore: registered new interface driver usbserial_generic [ 4.129399] usbserial: USB Serial support registered for generic [ 4.133379] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.139668] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.142527] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.146415] mousedev: PS/2 mouse device common for all mice [ 4.152205] rtc_cmos 00:05: RTC can wake from S4 [ 4.154402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.160470] rtc_cmos 00:05: registered as rtc0 [ 4.160534] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.160573] intel_pstate: CPU model not supported [ 4.166641] hid: raw HID events driver (C) Jiri Kosina [ 4.173521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.178363] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.181870] usbcore: registered new interface driver usbhid [ 4.181877] usbhid: USB HID core driver [ 4.182170] drop_monitor: Initializing network drop monitor service [ 4.183128] Initializing XFRM netlink socket [ 4.205342] NET: Registered protocol family 10 [ 4.209065] Segment Routing with IPv6 [ 4.210577] NET: Registered protocol family 17 [ 4.212942] mpls_gso: MPLS GSO support [ 4.221317] RAS: Correctable Errors collector initialized. [ 4.223830] AVX version of gcm_enc/dec engaged. [ 4.225887] AES CTR mode by8 optimization enabled [ 4.342625] sched_clock: Marking stable (4342519844, 0)->(5350917424, -1008397580) [ 4.346859] registered taskstats version 1 [ 4.349616] Loading compiled-in X.509 certificates [ 4.353123] zswap: loaded using pool lzo/zbud [ 4.387805] Key type big_key registered [ 4.409289] Key type encrypted registered [ 4.411871] ima: No TPM chip found, activating TPM-bypass! [ 4.416339] ima: Allocated hash algorithm: sha1 [ 4.419560] ima: No architecture policies found [ 4.421798] evm: Initialising EVM extended attributes: [ 4.425341] evm: security.selinux [ 4.426980] evm: security.ima [ 4.430386] evm: security.capability [ 4.431884] evm: HMAC attrs: 0x1 [ 4.437732] rtc_cmos 00:05: setting system clock to 2026-09-03 12:19:28 UTC (1788437968) [ 4.446882] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 4.454631] debug: unmapping init [mem 0xffffffff9f582000-0xffffffff9f858fff] [ 4.461066] Write protecting the kernel read-only data: 28672k [ 4.466788] debug: unmapping init [mem 0xffffffff9dc03000-0xffffffff9ddfffff] [ 4.471045] debug: unmapping init [mem 0xffffffff9e514000-0xffffffff9e5fffff] [ 4.541287] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.555665] systemd[1]: Detected virtualization kvm. [ 4.560092] systemd[1]: Detected architecture x86-64. [ 4.564245] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.608639] systemd[1]: No hostname configured. [ 4.613180] systemd[1]: Set hostname to . [ 4.617251] random: systemd: uninitialized urandom read (16 bytes read) [ 4.625196] systemd[1]: Initializing machine ID from random generator. [ 4.807429] random: ln: uninitialized urandom read (6 bytes read) [ 4.998693] random: systemd: uninitialized urandom read (16 bytes read) [ 5.004381] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.015389] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.031603] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.663053] device-mapper: uevent: version 1.0.3 [ 6.668348] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.808029] virtio_net virtio0 ens2: renamed from eth0 [ 8.871092] random: fast init done [ 9.116900] scsi host0: ata_piix [ 9.168522] scsi host1: ata_piix [ 9.174638] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.177535] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.953572] random: crng init done [ 13.955488] random: 7 urandom warning(s) missed due to ratelimiting [ 14.977500] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target R[ 17.289558] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) emote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.602586] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.234490] SELinux: Disabled at runtime. [ 21.345898] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.356574] systemd[1]: Detected virtualization kvm. [ 21.359277] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.157291] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.162237] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.177192] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.184120] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.193641] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.205894] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.220830] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ 23.512112] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 24.723515] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.949022] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.984276] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.515107] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.627854] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.876748] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.312173] NFS: Registering the id_resolver key type [ 32.313510] Key type id_resolver registered [ 32.314790] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-server login: [ 96.017355] libcfs: loading out-of-tree module taints kernel. [ 96.067521] Key type ._llcrypt registered [ 96.070874] Key type .llcrypt registered [ 96.152678] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 119.640943] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 121.300368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.334566] alg: No test for adler32 (adler32-zlib) [ 123.444954] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 124.775218] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 126.793435] Key type lgssc registered [ 129.865127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 150.879053] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.244035] hrtimer: interrupt took 10108277 ns [ 198.973613] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 213.644300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 213.674417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.958312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.031648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.215288] Lustre: lustre-MDT0000: new disk, initializing [ 215.356559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.408189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.156264] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.079167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.225702] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 235.297132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 235.310362] Lustre: Skipped 1 previous similar message [ 235.422881] Lustre: lustre-MDT0001: new disk, initializing [ 235.531099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.571727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 235.580859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 240.205870] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.183801] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 255.738391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.080120] Lustre: lustre-OST0000: new disk, initializing [ 256.086806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 256.094635] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 256.174333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.191302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 261.219935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 261.294652] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 262.905235] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.984582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 280.176397] Lustre: lustre-OST0001: new disk, initializing [ 280.179560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 280.186878] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 280.320314] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 286.822053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 286.839369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 286.955045] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 288.505791] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.199976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.373045] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 317.628398] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing check_logdir /tmp/testlogs/ [ 323.737888] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing yml_node [ 329.186933] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 332.323118] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 335.424791] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 337.571535] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:24:59 EDT 2026 [ 357.536680] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 359.923896] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 378.849133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 378.853073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.896858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 379.363419] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 382.682608] Lustre: server umount lustre-MDT0000 complete [ 387.568803] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438351 with bad export cookie 14487617308436225519 [ 387.571979] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 387.577746] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 389.602198] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.610049] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.618848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 389.618857] Lustre: Skipped 4 previous similar messages [ 389.623185] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 389.647950] Lustre: Skipped 3 previous similar messages [ 392.673202] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.673434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 392.695418] LustreError: 6525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 392.705921] Lustre: Skipped 2 previous similar messages [ 394.046987] Lustre: server umount lustre-MDT0001 complete [ 398.522500] Lustre: server umount lustre-OST0000 complete [ 402.552365] Lustre: server umount lustre-OST0001 complete [ 408.877764] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 420.970163] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 466.555402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.909471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.931781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 467.062256] Lustre: lustre-MDT0000: new disk, initializing [ 467.208680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.231584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 472.435756] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 484.760132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.963656] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 485.089451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 485.100489] Lustre: Skipped 1 previous similar message [ 485.238946] Lustre: lustre-MDT0001: new disk, initializing [ 485.403582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 485.471356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 485.498681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 490.409093] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 500.861887] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 507.204642] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.886369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 516.135303] Lustre: lustre-OST0000: new disk, initializing [ 516.139987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 516.151967] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 516.240980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.503618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 517.524588] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 517.660155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 523.868144] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 535.340881] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.808620] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 542.104412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 547.598404] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 547.878571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 553.446400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.451447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.452791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.459578] Lustre: Skipped 1 previous similar message [ 556.792344] Lustre: server umount lustre-OST0000 complete [ 562.163741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.174338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.190394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.197837] Lustre: Skipped 1 previous similar message [ 567.473842] Lustre: server umount lustre-MDT0000 complete [ 568.802485] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.820843] LustreError: 15097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 572.190093] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438536 with bad export cookie 14487617308436227052 [ 572.193151] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.202338] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.809986] Lustre: server umount lustre-MDT0001 complete [ 577.674795] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:28:59 (1788438539) [ 580.603188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 582.824698] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:29:04 (1788438544) [ 592.587790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.215139] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 593.337564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.501376] LustreError: 19465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.535165] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 608.739644] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.770189] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 610.732570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.288781] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 623.532895] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 629.409978] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 637.209538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.400223] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 654.420895] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 666.604313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.605527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 666.615979] Lustre: Skipped 2 previous similar messages [ 666.617232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 666.617238] Lustre: Skipped 2 previous similar messages [ 672.359573] Lustre: server umount lustre-OST0000 complete [ 677.861724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.644844] Lustre: server umount lustre-MDT0000 complete [ 687.073825] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 687.108209] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 688.009297] LustreError: 21307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438652 with bad export cookie 14487617308436227934 [ 688.018111] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 688.027047] LustreError: 21307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 688.569469] Lustre: server umount lustre-MDT0001 complete [ 701.965200] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 705.623504] Key type lgssc unregistered [ 705.948771] LNet: 23003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.962639] LNetError: 23003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.985984] LNet: Removed LNI 192.168.202.140@tcp [ 707.333514] Key type .llcrypt unregistered [ 707.335058] Key type ._llcrypt unregistered [ 733.766745] Key type ._llcrypt registered [ 733.771543] Key type .llcrypt registered [ 733.868725] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 735.314572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.375065] alg: No test for adler32 (adler32-zlib) [ 736.494326] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 736.766864] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 738.466801] Key type lgssc registered [ 739.969291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 749.720288] loop: module loaded [ 749.837513] loop0: detected capacity change from 0 to 204800000 [ 750.125083] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 750.140110] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 750.488772] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 757.182514] loop0: detected capacity change from 0 to 204800000 [ 757.457482] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 757.481220] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 757.502789] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 763.348675] loop0: detected capacity change from 0 to 204800000 [ 769.825384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 769.853071] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.235626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 771.304448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 771.444132] Lustre: lustre-MDT0000: new disk, initializing [ 771.545111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 771.572348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 776.882431] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 787.007145] loop1: detected capacity change from 0 to 204800000 [ 793.166791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 793.539681] Lustre: lustre-OST07c6: new disk, initializing [ 793.542936] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 793.548509] Lustre: Skipped 1 previous similar message [ 793.564133] Lustre: 26189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 793.659761] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 800.219574] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 800.915123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 800.927336] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 800.973400] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 804.971061] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 804.985454] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 804.991573] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 804.998940] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 805.011921] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 805.017360] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 822.054285] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 822.085501] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 822.096786] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 822.118332] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 822.151197] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 822.169581] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 827.593821] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 827.600147] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 827.604400] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 827.610575] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 827.617099] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 827.620957] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 827.625666] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 827.635663] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 827.641845] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 827.645074] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 827.648473] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 827.652026] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.760440] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 832.771195] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 832.784571] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 832.793688] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.806377] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 832.816315] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.823906] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 832.837466] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.846786] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 832.858598] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 832.872898] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 832.886268] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.557582] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 838.580860] Lustre: 25176:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 838.596879] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 838.604541] Lustre: 25176:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.622245] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 838.630924] Lustre: 25176:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.640609] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 838.658645] Lustre: 25176:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.677082] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 838.683949] Lustre: 25176:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 838.691470] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 838.701755] Lustre: 25176:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 847.236462] Lustre: 26886:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 847.254406] Lustre: 26886:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 847.262861] Lustre: 26886:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 847.267045] Lustre: 26886:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.271185] Lustre: 26886:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 847.273944] Lustre: 26886:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.276891] Lustre: 26886:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 847.280992] Lustre: 26886:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.284905] Lustre: 26886:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 847.289065] Lustre: 26886:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 847.293916] Lustre: 26886:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 847.298597] Lustre: 26886:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 903.152658] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 903.177182] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 907.510400] Lustre: server umount lustre-OST07c6 complete [ 918.809240] Lustre: server umount lustre-MDT0000 complete [ 939.718568] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 943.039800] Key type lgssc unregistered [ 943.456742] LNet: 28750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.467357] LNetError: 28750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.492506] LNet: Removed LNI 192.168.202.140@tcp [ 944.437502] Key type .llcrypt unregistered [ 944.439299] Key type ._llcrypt unregistered [ 969.801305] Key type ._llcrypt registered [ 969.805227] Key type .llcrypt registered [ 969.913884] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 970.974425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.106996] alg: No test for adler32 (adler32-zlib) [ 972.367290] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 972.661611] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 974.361464] Key type lgssc registered [ 975.935375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.045840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.081745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.731944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.262234] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 997.864084] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.890980] LustreError: 30020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1002.991178] LustreError: 30019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1003.302813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.959226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.983807] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1019.878130] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1027.769789] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1036.625143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.170223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.022582] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1053.842899] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1057.661473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1078.244940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.256515] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.290668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.272784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.282893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.298405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.326705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.342365] Lustre: Skipped 1 previous similar message [ 1083.728566] Lustre: server umount lustre-OST0000 complete [ 1091.041892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1091.055798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.083196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.676404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1096.692375] Lustre: Skipped 2 previous similar messages [ 1103.328352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1103.558915] Lustre: server umount lustre-MDT0000 complete [ 1106.915217] LustreError: 30014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.939161] LustreError: 30014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1108.793078] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439072 with bad export cookie 14875669938008205478 [ 1108.799537] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1108.816881] LustreError: 29999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1109.305802] Lustre: server umount lustre-MDT0001 complete [ 1121.165128] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1124.750117] Key type lgssc unregistered [ 1125.119251] LNet: 33778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.128019] LNetError: 33778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.147857] LNet: Removed LNI 192.168.202.140@tcp [ 1126.350437] Key type .llcrypt unregistered [ 1126.353564] Key type ._llcrypt unregistered [ 1150.101695] Key type ._llcrypt registered [ 1150.108150] Key type .llcrypt registered [ 1150.303127] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1152.371610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1152.402876] alg: No test for adler32 (adler32-zlib) [ 1153.648386] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1154.004293] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1155.808249] Key type lgssc registered [ 1157.351103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.577916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.592714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.135668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.058922] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1180.662275] LustreError: 35048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.687973] LustreError: 35048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1184.856721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.259263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.220160] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1198.581848] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1205.703583] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.467380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.004603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.064988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1221.952419] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1231.204546] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.669270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1245.669773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.675461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1245.687984] LustreError: Skipped 1 previous similar message [ 1245.712275] Lustre: Skipped 1 previous similar message [ 1247.953682] Lustre: server umount lustre-OST0000 complete [ 1255.916195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.924407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.939664] Lustre: Skipped 1 previous similar message [ 1255.947636] Lustre: Skipped 1 previous similar message [ 1258.963273] Lustre: server umount lustre-MDT0000 complete [ 1261.027978] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.047225] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1263.253501] LustreError: 35027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439227 with bad export cookie 9737153073051062154 [ 1263.258844] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.263080] LustreError: 35027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1263.722301] Lustre: server umount lustre-MDT0001 complete [ 1278.018717] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1281.773872] Key type lgssc unregistered [ 1282.070066] LNet: 38582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.079416] LNetError: 38582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.106858] LNet: Removed LNI 192.168.202.140@tcp [ 1283.290552] Key type .llcrypt unregistered [ 1283.295723] Key type ._llcrypt unregistered [ 1312.622161] Key type ._llcrypt registered [ 1312.625786] Key type .llcrypt registered [ 1312.741024] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1314.345464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1314.505867] alg: No test for adler32 (adler32-zlib) [ 1315.762635] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1316.062472] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1317.780256] Key type lgssc registered [ 1319.560898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.548982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.582087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.057371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.251941] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1341.426280] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.446498] LustreError: 40000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1346.536365] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.445678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.762980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.865476] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1362.800630] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1371.437898] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1380.056045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1380.300735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.184883] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1393.615904] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.803400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1412.082227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.104240] Lustre: Skipped 1 previous similar message [ 1412.108080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1412.132724] Lustre: Skipped 2 previous similar messages [ 1415.011993] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1416.699253] Lustre: server umount lustre-MDT0000 complete [ 1417.190311] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1417.219748] LustreError: 39999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1420.099532] LustreError: 39995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.951325] LustreError: 39979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439385 with bad export cookie 8414960792199858110 [ 1420.972780] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.314126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.327167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1425.226073] LustreError: 39996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.242241] LustreError: 39996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1425.249380] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 1427.624754] Lustre: server umount lustre-MDT0001 complete [ 1458.145758] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1458.389629] Lustre: server umount lustre-OST0000 complete [ 1480.343081] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1484.421654] Key type lgssc unregistered [ 1484.942794] LNet: 43882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.954427] LNetError: 43882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.008127] LNet: Removed LNI 192.168.202.140@tcp [ 1486.400252] Key type .llcrypt unregistered [ 1486.405984] Key type ._llcrypt unregistered [ 1514.149153] Key type ._llcrypt registered [ 1514.150633] Key type .llcrypt registered [ 1514.233570] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1515.590310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.754235] alg: No test for adler32 (adler32-zlib) [ 1516.898707] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1517.199860] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1518.873809] Key type lgssc registered [ 1520.335857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1530.146576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1530.166296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.782221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.865521] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1541.112411] LustreError: 45296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.124741] LustreError: 45296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1546.212631] LustreError: 45297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1549.302635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.851663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1556.057983] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1566.109970] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1574.320158] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1583.058706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1583.605811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.258307] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1601.110543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1601.491341] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1608.558633] Lustre: lustre-OST0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1610.210122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1610.227231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.244920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.235661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.259278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1611.261782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1611.293399] Lustre: Skipped 1 previous similar message [ 1613.642759] Lustre: lustre-OST0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1614.549990] Lustre: server umount lustre-OST0000 complete [ 1636.326391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.337116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.356399] Lustre: Skipped 1 previous similar message [ 1636.381737] Lustre: Skipped 1 previous similar message [ 1640.607052] Lustre: server umount lustre-MDT0000 complete [ 1641.443736] LustreError: 45296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1641.462436] LustreError: 45296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1644.630424] LustreError: 45276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439608 with bad export cookie 11733836711313033228 [ 1644.644102] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1651.170668] Lustre: server umount lustre-MDT0001 complete [ 1663.680892] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1667.755514] Key type lgssc unregistered [ 1668.205186] LNet: 49023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.216062] LNetError: 49023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.235141] LNet: Removed LNI 192.168.202.140@tcp [ 1669.444782] Key type .llcrypt unregistered [ 1669.449616] Key type ._llcrypt unregistered [ 1698.665309] Key type ._llcrypt registered [ 1698.668376] Key type .llcrypt registered [ 1698.896422] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1699.759569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.777120] alg: No test for adler32 (adler32-zlib) [ 1701.008091] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1701.288789] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1703.155929] Key type lgssc registered [ 1705.348618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1718.360238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1718.448332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.455816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.507235] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1730.020159] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.049455] LustreError: 50434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1735.141316] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.143377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.509249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1742.779218] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1753.256462] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1761.919494] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1771.602726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1772.121896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.050705] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1790.330952] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.533608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1796.932119] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1801.073792] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1814.367896] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h49m35s [ 1816.409769] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1821.154082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1821.157348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1821.178710] Lustre: Skipped 1 previous similar message [ 1821.212521] Lustre: Skipped 2 previous similar messages [ 1822.243807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1823.572458] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 1823.575883] Lustre: Skipped 2 previous similar messages [ 1825.672542] Lustre: server umount lustre-MDT0000 complete [ 1828.336519] LustreError: 51360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1828.359341] LustreError: 51360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1830.727318] LustreError: 50431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.834594] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1836.866928] LustreError: 50430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.895551] LustreError: 50430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1840.036120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.153951] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1840.548588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.693145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1841.591222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1846.459605] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1850.652951] Lustre: DEBUG MARKER: Wait for df (36778) ... [ 1852.618856] Lustre: DEBUG MARKER: done [ 1859.042026] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.046041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.051380] Lustre: Skipped 1 previous similar message [ 1859.056127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.069720] Lustre: Skipped 1 previous similar message [ 1861.091664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1861.107645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.238986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.268309] Lustre: Skipped 4 previous similar messages [ 1871.328913] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1871.490940] Lustre: server umount lustre-OST0000 complete [ 1877.473814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1877.491241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.508794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.548537] Lustre: Skipped 1 previous similar message [ 1882.846397] Lustre: server umount lustre-MDT0000 complete [ 1884.645304] LustreError: 51360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1884.678980] LustreError: 51360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1887.038651] LustreError: 50416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439851 with bad export cookie 1747368535928024800 [ 1887.042918] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.056781] LustreError: 50416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1887.820414] Lustre: server umount lustre-MDT0001 complete [ 1900.027420] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 1903.548910] Key type lgssc unregistered [ 1904.006158] LNet: 55365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.020225] LNetError: 55365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.053511] LNet: Removed LNI 192.168.202.140@tcp [ 1905.116287] Key type .llcrypt unregistered [ 1905.119244] Key type ._llcrypt unregistered [ 1951.899892] Key type ._llcrypt registered [ 1951.904701] Key type .llcrypt registered [ 1952.067814] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 1953.883629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.993271] alg: No test for adler32 (adler32-zlib) [ 1955.056130] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1955.345962] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 1957.056192] Key type lgssc registered [ 1958.233620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1967.068909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1967.086504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1968.309601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1968.334961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1968.440174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.533149] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1981.672563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.823551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1981.851546] Lustre: 57573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1982.002314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1982.108885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.462743] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 1996.070706] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2003.181432] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.351859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2010.642344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2010.886901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.417535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2017.936650] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2026.765738] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2035.525863] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h53m16s [ 2037.789077] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2046.303883] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2049.732270] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2049.777256] LustreError: 59442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2069.895855] Lustre: lustre-MDT0000: Client 068c2b0a-8fbd-44ed-852b-debd8d705b41 (at 192.168.202.40@tcp) reconnecting [ 2089.816120] LustreError: 59442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2092.510467] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2095.049341] Lustre: DEBUG MARKER: done [ 2104.290180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.305879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.340744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2104.801685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2104.828064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.636832] Lustre: server umount lustre-OST0000 complete [ 2115.048077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.057899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2115.063122] Lustre: Skipped 1 previous similar message [ 2115.070588] Lustre: Skipped 2 previous similar messages [ 2122.220937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2122.227403] Lustre: Skipped 3 previous similar messages [ 2127.333914] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2127.620709] Lustre: server umount lustre-MDT0000 complete [ 2132.194987] LustreError: 57566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440096 with bad export cookie 13722723733308077917 [ 2132.201500] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.213844] LustreError: 57566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2132.670691] Lustre: server umount lustre-MDT0001 complete [ 2144.232977] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2147.299575] Key type lgssc unregistered [ 2147.892797] LNet: 61905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.914720] LNetError: 61905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.960844] LNet: Removed LNI 192.168.202.140@tcp [ 2149.474527] Key type .llcrypt unregistered [ 2149.481361] Key type ._llcrypt unregistered [ 2207.709857] Key type ._llcrypt registered [ 2207.711939] Key type .llcrypt registered [ 2207.820672] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2209.353389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.376676] alg: No test for adler32 (adler32-zlib) [ 2210.414612] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2210.762863] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2212.424480] Key type lgssc registered [ 2213.793766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.831714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.857777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.227132] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2218.285479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2218.561639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2219.721202] Lustre: Failing over lustre-MDT0000 [ 2220.030426] Lustre: server umount lustre-MDT0000 complete [ 2227.304230] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:56:29 (1788440189) [ 2234.557490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.940453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.568144] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2250.912849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.112741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2251.169892] Lustre: 64704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2251.580041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.336744] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2266.091300] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.914467] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2281.283636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2281.815511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2282.229444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.974464] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2292.005157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2302.168864] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.550117] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2317.811866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.828611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2322.055253] Lustre: server umount lustre-MDT0000 complete [ 2326.914074] LustreError: 65440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440290 with bad export cookie 13283318031107049705 [ 2326.920338] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2326.934522] LustreError: 65440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2327.384169] Lustre: server umount lustre-MDT0001 complete [ 2331.541879] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2336.401023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2344.161061] Lustre: 63627:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440292/real 1788440292] req@000000000bf5f8e8 x1875315443935488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440308 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2344.200743] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2344.217094] Lustre: Skipped 2 previous similar messages [ 2345.509038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2347.488354] Lustre: 63625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440295/real 1788440295] req@000000003c893c88 x1875315443935616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2349.602965] Lustre: 63625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440297/real 1788440297] req@00000000ea9bab8e x1875315443935744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440313 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2352.863038] LustreError: 66556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.888238] LustreError: 66556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2352.964742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.091460] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2353.634499] Lustre: 63626:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440301/real 1788440301] req@00000000ab500354 x1875315443936000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2353.962346] LustreError: 66557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.983949] LustreError: 66557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2358.270959] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2359.265818] Lustre: 63626:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440306/real 1788440306] req@000000001e5b1c77 x1875315443936128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440322 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2359.274877] LustreError: 66556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.346269] LustreError: 66556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2364.386778] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.412982] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2368.564945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.151373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2373.316513] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2374.796863] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2383.885850] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.567280] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2404.325068] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.331383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.345418] Lustre: Skipped 1 previous similar message [ 2404.357430] Lustre: Skipped 2 previous similar messages [ 2406.383140] Lustre: server umount lustre-MDT0000 complete [ 2409.446312] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.477650] LustreError: 68477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2410.965317] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440375 with bad export cookie 13283318031107051980 [ 2410.976757] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.981243] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2411.324598] Lustre: server umount lustre-MDT0001 complete [ 2418.841995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2429.426836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.944239] Lustre: 63627:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440378/real 1788440378] req@00000000f96451f9 x1875315443965056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2430.947314] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.986595] Lustre: 63627:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2431.044343] Lustre: Skipped 1 previous similar message [ 2436.196884] LustreError: 63623:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000081d23ab7 x1875315443966080/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2436.604868] LustreError: 66557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2436.686123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2436.778477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2442.232428] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2451.512942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2457.078597] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2457.088682] Lustre: Skipped 1 previous similar message [ 2457.182306] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2459.194271] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2466.149173] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2472.974375] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2482.658724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2482.663989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.680120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.685141] Lustre: Skipped 2 previous similar messages [ 2484.716541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.738394] Lustre: Skipped 2 previous similar messages [ 2486.952667] Lustre: server umount lustre-MDT0000 complete [ 2489.825920] LustreError: 71894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.853096] LustreError: 71894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2491.227315] LustreError: 64697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440455 with bad export cookie 13283318031107053359 [ 2491.241328] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.828904] Lustre: server umount lustre-MDT0001 complete [ 2500.217881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2504.385437] Lustre: DEBUG MARKER: files compared the same [ 2516.318606] Lustre: server umount lustre-OST0000 complete [ 2536.049750] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2539.275892] Key type lgssc unregistered [ 2539.573753] LNet: 74453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.577460] LNetError: 74453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2539.604611] LNet: Removed LNI 192.168.202.140@tcp [ 2540.672253] Key type .llcrypt unregistered [ 2540.674464] Key type ._llcrypt unregistered [ 2566.935674] Key type ._llcrypt registered [ 2566.938354] Key type .llcrypt registered [ 2567.132888] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2568.888837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2568.917431] alg: No test for adler32 (adler32-zlib) [ 2570.128065] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2570.480302] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2572.224223] Key type lgssc registered [ 2573.748327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.686580] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.704404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.340404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2587.807961] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug +malloc all [ 2593.781308] LustreError: 75867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.796591] LustreError: 75867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2597.168764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.797549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2604.706648] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug +malloc all [ 2613.428231] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2620.401739] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2628.198084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2628.421428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.481493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2635.712755] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug +malloc all [ 2645.280160] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2655.204572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.207874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.231045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.277337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2658.281474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.281837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.290450] Lustre: server umount lustre-OST0000 complete [ 2668.513325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.552236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.576538] Lustre: Skipped 1 previous similar message [ 2672.819298] Lustre: server umount lustre-MDT0000 complete [ 2673.637131] LustreError: 75868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.665167] LustreError: 75868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2677.341244] LustreError: 75847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440641 with bad export cookie 8633714262905708514 [ 2677.358124] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.374864] LustreError: 75847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2677.939880] Lustre: server umount lustre-MDT0001 complete [ 2690.901606] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2694.914563] Key type lgssc unregistered [ 2695.131767] LNet: 79399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.137624] LNetError: 79399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.155231] LNet: Removed LNI 192.168.202.140@tcp [ 2696.203591] Key type .llcrypt unregistered [ 2696.214484] Key type ._llcrypt unregistered [ 2727.312097] Key type ._llcrypt registered [ 2727.314101] Key type .llcrypt registered [ 2727.463166] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2729.188349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.481816] alg: No test for adler32 (adler32-zlib) [ 2730.699856] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2731.105301] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2732.850833] Key type lgssc registered [ 2734.324136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.529310] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2743.570112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2759.200800] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2764.321535] LustreError: 80787:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [ 2791.008377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.903778] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2807.778665] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2816.924242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.108841] Lustre: *** cfs_fail_loc=706, val=0*** [ 2817.615905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.759135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2822.347809] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2822.626256] LustreError: 80813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.646798] LustreError: 80813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2825.715601] LustreError: 81873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.733838] LustreError: 81873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2830.845245] LustreError: 81874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.864903] LustreError: 81874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2832.292665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.700652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.353510] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2847.444188] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2854.697420] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2858.499755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.499895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.516783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2858.531468] LustreError: Skipped 1 previous similar message [ 2858.557791] Lustre: Skipped 1 previous similar message [ 2863.485954] Lustre: server umount lustre-OST0000 complete [ 2868.711038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.722319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.737736] Lustre: Skipped 1 previous similar message [ 2868.739035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.851746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2873.865311] Lustre: Skipped 2 previous similar messages [ 2874.200942] Lustre: server umount lustre-MDT0000 complete [ 2877.995330] LustreError: 81878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.031649] LustreError: 81878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2878.410162] LustreError: 80793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440842 with bad export cookie 9159000739243018049 [ 2878.413062] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2878.428052] LustreError: 80793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2878.723625] Lustre: server umount lustre-MDT0001 complete [ 2889.622128] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 2892.908907] Key type lgssc unregistered [ 2893.359530] LNet: 84405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.363796] LNetError: 84405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.379794] LNet: Removed LNI 192.168.202.140@tcp [ 2894.391230] Key type .llcrypt unregistered [ 2894.393178] Key type ._llcrypt unregistered [ 2922.582527] Key type ._llcrypt registered [ 2922.584825] Key type .llcrypt registered [ 2922.694447] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 2923.944142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2924.154088] alg: No test for adler32 (adler32-zlib) [ 2926.030527] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2926.390799] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 2928.192148] Key type lgssc registered [ 2929.070726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.407736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.445697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.522425] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2954.559570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.214547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2960.366910] LustreError: 86508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.411803] LustreError: 86508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2962.528558] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2965.476161] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.513784] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2970.600818] LustreError: 86511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.635101] LustreError: 86511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2971.931692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2972.495310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2977.762017] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.785279] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2979.461437] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 2985.955559] LustreError: 86508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.971379] LustreError: 86508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2988.838484] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.858307] Lustre: MGS: Client b7fac129-0bc4-4936-b3b6-94ff2f68a202 (at 0@lo) reconnecting [ 2988.887564] LustreError: 86522:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2989.235312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.337145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2994.458402] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3010.532274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3010.547814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.591412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.145103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3015.149370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3015.149660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.154086] Lustre: Skipped 1 previous similar message [ 3016.723503] Lustre: server umount lustre-OST0000 complete [ 3025.386774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.388226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3025.398505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3027.251503] Lustre: server umount lustre-MDT0000 complete [ 3030.500125] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.510022] LustreError: 86507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3032.266470] Lustre: server umount lustre-MDT0001 complete [ 3036.491311] Lustre: server umount MGS complete [ 3050.190574] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:10:11 (1788441011) [ 3085.912542] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 3098.859225] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3141.159760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.672318] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3160.521857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.985522] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3161.059832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3161.209515] Lustre: lustre-MDT0001: new disk, initializing [ 3161.334028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.402538] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3161.421398] Lustre: 93701:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3161.436267] LustreError: 93701:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3168.960925] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3183.130881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.660336] Lustre: lustre-OST0000: new disk, initializing [ 3183.663391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3183.668533] Lustre: 94469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3183.782469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.905121] Lustre: 94710:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000020510d18 x1875316194716160/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788441171 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3192.220390] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3203.999513] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3212.270049] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3213.512810] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3213.525603] Lustre: MGS: Client 62453c55-b5ce-4a5b-9b82-26f5db10dce6 (at 0@lo) reconnecting [ 3213.596172] Lustre: 92968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3213.631050] LustreError: 92968:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 3213.643852] LustreError: 92968:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3213.695314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3213.884502] Lustre: lustre-MDT0000: new disk, initializing [ 3214.090782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3214.128977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3215.210991] Lustre: 95540:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b3897032 x1875316194723968/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788441195 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3219.203676] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3220.261900] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3224.303804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3224.315180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3224.469432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3251.701310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.716386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3251.720267] Lustre: Skipped 2 previous similar messages [ 3251.744271] Lustre: Skipped 2 previous similar messages [ 3256.815551] Lustre: server umount lustre-OST0000 complete [ 3261.927777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3261.928338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.943990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3261.961596] Lustre: Skipped 1 previous similar message [ 3261.972034] Lustre: Skipped 1 previous similar message [ 3267.498968] Lustre: server umount lustre-MDT0000 complete [ 3271.142205] LustreError: 95851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.688784] Lustre: server umount lustre-MDT0001 complete [ 3283.301828] Lustre: server umount MGS complete [ 3296.787401] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:14:18 (1788441258) [ 3318.748473] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3322.957658] Key type lgssc unregistered [ 3323.436831] LNet: 98039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.448948] LNetError: 98039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3323.477057] LNet: Removed LNI 192.168.202.140@tcp [ 3324.759911] Key type .llcrypt unregistered [ 3324.763718] Key type ._llcrypt unregistered [ 3342.661906] Key type ._llcrypt registered [ 3342.665781] Key type .llcrypt registered [ 3342.838942] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3343.964181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.992269] alg: No test for adler32 (adler32-zlib) [ 3345.412487] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3345.633103] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3347.376514] Key type lgssc registered [ 3348.516544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3356.627398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3356.643471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.252734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3363.964835] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3368.430205] LustreError: 99257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.456465] LustreError: 99257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3373.540099] LustreError: 99256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3374.007314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.610686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.532863] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3389.998735] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3396.342431] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3404.970298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3405.442107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3413.366783] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3423.443244] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3427.974122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3438.561417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.576312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.589678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3438.606309] Lustre: Skipped 1 previous similar message [ 3443.688640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3443.697994] Lustre: Skipped 1 previous similar message [ 3444.267343] Lustre: server umount lustre-OST0000 complete [ 3448.803702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.807040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3448.818705] Lustre: Skipped 2 previous similar messages [ 3448.842721] Lustre: Skipped 1 previous similar message [ 3453.926056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.928860] Lustre: Skipped 1 previous similar message [ 3455.013793] Lustre: server umount lustre-MDT0000 complete [ 3458.990389] LustreError: 99238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441423 with bad export cookie 1290107475259188178 [ 3459.007029] LustreError: 99238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3459.010651] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3459.363889] Lustre: server umount lustre-MDT0001 complete [ 3471.764216] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3475.133056] Key type lgssc unregistered [ 3475.479903] LNet: 102812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3475.486352] LNetError: 102812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3475.501934] LNet: Removed LNI 192.168.202.140@tcp [ 3476.607668] Key type .llcrypt unregistered [ 3476.610093] Key type ._llcrypt unregistered [ 3493.785996] Key type ._llcrypt registered [ 3493.789220] Key type .llcrypt registered [ 3493.924829] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3495.165493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3495.201534] alg: No test for adler32 (adler32-zlib) [ 3496.487783] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3496.754140] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3498.408399] Key type lgssc registered [ 3499.603353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3505.830175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3505.834307] LustreError: 103893:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3505.889837] LustreError: 103898:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3505.904341] LustreError: 103893:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4930 [ 3505.914596] LustreError: 103893:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3505.920715] LustreError: 103893:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3505.929122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.365959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3515.434740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3515.740603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3520.382206] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3527.304706] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing lsmod [ 3530.923550] LustreError: 105104:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3530.927231] LustreError: 105103:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3530.949312] LustreError: 105104:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4981 [ 3530.961294] LustreError: 105104:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3530.972157] LustreError: 105104:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3530.979417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3531.425132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.504397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3536.359735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3536.369571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.386749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.890554] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.897951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.928921] Lustre: Skipped 1 previous similar message [ 3537.949078] Lustre: Skipped 2 previous similar messages [ 3541.034828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3541.058023] Lustre: Skipped 2 previous similar messages [ 3546.087606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.104967] Lustre: Skipped 2 previous similar messages [ 3549.153182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3549.394214] Lustre: server umount lustre-MDT0000 complete [ 3551.207714] LustreError: 103936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.230442] LustreError: 103936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3553.422270] LustreError: 103910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441517 with bad export cookie 15512209096566520905 [ 3553.426200] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3553.431068] LustreError: 103910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3555.424465] LustreError: 104367:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004b79687b x1875316792385280/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3555.576493] Lustre: server umount lustre-MDT0001 complete [ 3569.460354] Lustre: server umount lustre-OST0000 complete [ 3578.172335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.717790] LustreError: 106170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.811610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3583.970128] LustreError: 106171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3584.000194] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3588.078751] LustreError: 106170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.187626] LustreError: 106171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.314160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.117138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3602.541882] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3612.653268] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.359838] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.206777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.804585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3634.721355] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3643.780600] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3648.263055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3658.721373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.723274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.750182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.759334] Lustre: Skipped 1 previous similar message [ 3658.777159] Lustre: Skipped 1 previous similar message [ 3663.081931] Lustre: server umount lustre-OST0000 complete [ 3668.963869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3668.967049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3668.976920] Lustre: Skipped 1 previous similar message [ 3673.788205] Lustre: server umount lustre-MDT0000 complete [ 3674.102312] LustreError: 106171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.143401] LustreError: 106171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3677.667229] LustreError: 106151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441641 with bad export cookie 15512209096566521850 [ 3677.679866] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3677.683245] LustreError: 106151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3678.052947] Lustre: server umount lustre-MDT0001 complete [ 3692.785057] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3695.898305] Key type lgssc unregistered [ 3696.188957] LNet: 109717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3696.196964] LNetError: 109717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3696.209891] LNet: Removed LNI 192.168.202.140@tcp [ 3697.003523] Key type .llcrypt unregistered [ 3697.008734] Key type ._llcrypt unregistered [ 3725.061566] Key type ._llcrypt registered [ 3725.063569] Key type .llcrypt registered [ 3725.213519] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3727.019470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3727.047701] alg: No test for adler32 (adler32-zlib) [ 3728.197877] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3728.518406] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3730.192314] Key type lgssc registered [ 3731.636148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3740.559995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3740.586173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.230201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3747.563807] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3752.418177] LustreError: 111128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3752.448849] LustreError: 111128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3757.538378] LustreError: 111129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.531664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3758.936978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3764.388161] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3774.458386] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3781.561937] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3791.033189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3791.605799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3799.387628] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3808.729411] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3812.089258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3818.512599] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3824.302488] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3832.809413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.830869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.836022] Lustre: Skipped 1 previous similar message [ 3832.852504] Lustre: Skipped 1 previous similar message [ 3836.361087] Lustre: server umount lustre-OST0000 complete [ 3840.994958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3840.999187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.007892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.042040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3843.046899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3843.063391] Lustre: Skipped 1 previous similar message [ 3846.944344] Lustre: server umount lustre-MDT0000 complete [ 3848.161602] LustreError: 111125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.191408] LustreError: 111125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3851.723871] LustreError: 111110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441815 with bad export cookie 2036584133271013106 [ 3851.737889] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3851.740275] LustreError: 111110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3852.152233] Lustre: server umount lustre-MDT0001 complete [ 3866.249638] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 3869.576700] Key type lgssc unregistered [ 3869.930323] LNet: 114770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3869.939675] LNetError: 114770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3871.019375] LNet: Removed LNI 192.168.202.140@tcp [ 3872.311589] Key type .llcrypt unregistered [ 3872.316869] Key type ._llcrypt unregistered [ 3893.267832] Key type ._llcrypt registered [ 3893.270840] Key type .llcrypt registered [ 3893.408964] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 3895.037367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3895.286449] alg: No test for adler32 (adler32-zlib) [ 3896.395813] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3896.964467] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 3898.811650] Key type lgssc registered [ 3900.063884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3908.243993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3908.275757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.969775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.536734] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3919.333059] LustreError: 115989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3919.376472] LustreError: 115989:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3924.460278] LustreError: 115990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.570550] LustreError: 115989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3930.096522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.648804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.537635] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3945.262636] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3952.776373] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3960.672543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3960.976842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3967.473937] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 3975.905987] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3979.446806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3993.074132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.082907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.096092] Lustre: Skipped 1 previous similar message [ 3993.134679] Lustre: Skipped 1 previous similar message [ 3997.667429] Lustre: server umount lustre-OST0000 complete [ 4002.277839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4002.285204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.296715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.297995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4003.301213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.424299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.427122] Lustre: Skipped 1 previous similar message [ 4008.625856] Lustre: server umount lustre-MDT0000 complete [ 4013.538128] LustreError: 118483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.662230] LustreError: 116724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441977 with bad export cookie 15803276388511700707 [ 4013.687480] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.695705] LustreError: 116724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4014.360353] Lustre: server umount lustre-MDT0001 complete [ 4027.628524] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 4030.825574] Key type lgssc unregistered [ 4031.155935] LNet: 119545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4031.160785] LNetError: 119545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4031.178468] LNet: Removed LNI 192.168.202.140@tcp [ 4032.097972] Key type .llcrypt unregistered [ 4032.100329] Key type ._llcrypt unregistered [ 4058.062450] Key type ._llcrypt registered [ 4058.067187] Key type .llcrypt registered [ 4058.208197] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4059.643138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4059.693403] alg: No test for adler32 (adler32-zlib) [ 4061.090507] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4061.478122] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 4063.208371] Key type lgssc registered [ 4064.674642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4072.949195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4072.966777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.451669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4079.219637] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4084.714220] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.723957] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4088.755077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.384618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.908589] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4103.986488] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4110.651928] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4117.890205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4118.416498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.421774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4125.437277] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4134.484826] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.855681] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4145.703914] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4145.707701] Lustre: Skipped 2 previous similar messages [ 4147.799691] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4147.849715] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4150.525914] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4150.534187] Lustre: Skipped 2 previous similar messages [ 4159.488104] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4159.491534] Lustre: Skipped 1 previous similar message [ 4165.170042] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4165.176784] Lustre: Skipped 2 previous similar messages [ 4165.545171] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.40@tcp 192.168.0.[2,10]@tcp [ 4179.326207] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.40/24@tcp in log lustre-MDT0000 [ 4179.343791] Lustre: Skipped 2 previous similar messages [ 4187.698949] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.40/24@tcp [ 4187.705569] Lustre: Skipped 1 previous similar message [ 4196.322719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.326900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.327556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.341472] Lustre: Skipped 1 previous similar message [ 4201.448699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4201.455685] Lustre: Skipped 2 previous similar messages [ 4201.756709] Lustre: server umount lustre-OST0000 complete [ 4206.562696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.564840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4206.582292] Lustre: Skipped 1 previous similar message [ 4206.601482] Lustre: Skipped 1 previous similar message [ 4211.681371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4211.692663] Lustre: Skipped 1 previous similar message [ 4216.801860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.810083] Lustre: Skipped 1 previous similar message [ 4220.384192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4220.570550] Lustre: server umount lustre-MDT0000 complete [ 4221.926784] LustreError: 120952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4221.967160] LustreError: 120952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4224.882451] LustreError: 120937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442188 with bad export cookie 4747509442254798543 [ 4224.889358] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4224.896709] LustreError: 120937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4225.185838] Lustre: server umount lustre-MDT0001 complete [ 4237.764743] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 4241.294732] Key type lgssc unregistered [ 4241.699787] LNet: 125969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4241.719336] LNetError: 125969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4242.791162] LNet: Removed LNI 192.168.202.140@tcp [ 4243.821440] Key type .llcrypt unregistered [ 4243.823401] Key type ._llcrypt unregistered [ 4279.920619] Key type ._llcrypt registered [ 4279.923281] Key type .llcrypt registered [ 4280.060373] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4281.017150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4281.043188] alg: No test for adler32 (adler32-zlib) [ 4282.276637] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4282.469041] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 4284.225291] Key type lgssc registered [ 4285.580993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4294.550742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4294.583154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.143205] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4296.167128] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.40/24@tcp [ 4296.185171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.148329] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4306.402548] LustreError: 127529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4306.422016] LustreError: 127529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4310.179828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.442452] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4310.479959] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.40/24@tcp [ 4310.514417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4314.825511] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4323.258882] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4330.053537] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4337.531883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4337.976579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.066447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4344.133087] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4352.756355] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4366.820092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4366.836437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.857535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.917208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.918427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4370.919212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.928242] Lustre: Skipped 1 previous similar message [ 4372.971628] Lustre: server umount lustre-OST0000 complete [ 4381.153316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.157262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.171396] Lustre: Skipped 1 previous similar message [ 4383.646941] Lustre: server umount lustre-MDT0000 complete [ 4386.275410] LustreError: 127528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.292579] LustreError: 127528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4387.858845] LustreError: 127509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442351 with bad export cookie 14450724981139573280 [ 4387.879103] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.190932] Lustre: server umount lustre-MDT0001 complete [ 4401.526687] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 4404.842546] Key type lgssc unregistered [ 4405.246709] LNet: 131116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4405.260126] LNetError: 131116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4405.280669] LNet: Removed LNI 192.168.202.140@tcp [ 4406.359699] Key type .llcrypt unregistered [ 4406.362971] Key type ._llcrypt unregistered [ 4438.395958] Key type ._llcrypt registered [ 4438.397497] Key type .llcrypt registered [ 4438.525472] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 4456.765670] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4457.925821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4458.005487] alg: No test for adler32 (adler32-zlib) [ 4459.067878] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4459.353103] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 4461.096190] Key type lgssc registered [ 4462.333291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4506.824182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4506.872462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.236707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4508.303818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4508.380195] Lustre: lustre-MDT0000: new disk, initializing [ 4508.456811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.507488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4514.033293] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4526.617500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4526.755538] Lustre: 134518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4526.850739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4526.856380] Lustre: Skipped 1 previous similar message [ 4526.972309] Lustre: lustre-MDT0001: new disk, initializing [ 4527.076959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4527.115510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4527.137671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4531.842276] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4541.563226] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4547.965299] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4555.177782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.660681] Lustre: lustre-OST0000: new disk, initializing [ 4555.664548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4555.678541] Lustre: 136485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4555.797567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4561.985669] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4563.007593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4563.013626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4563.176110] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4571.249427] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4575.625414] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4575.848291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4580.109306] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4580.326335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4583.905388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.909047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.911959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.928816] Lustre: Skipped 1 previous similar message [ 4588.246444] Lustre: server umount lustre-OST0000 complete [ 4593.633023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4593.640389] LustreError: Skipped 1 previous similar message [ 4593.646063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.664222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4593.668459] Lustre: Skipped 1 previous similar message [ 4598.623213] Lustre: server umount lustre-MDT0000 complete [ 4602.372102] LustreError: 138026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442566 with bad export cookie 13529763791533486641 [ 4602.383248] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4602.390407] LustreError: 138026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4602.735987] Lustre: server umount lustre-MDT0001 complete [ 4611.663630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.135424] LustreError: 138702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4612.146467] LustreError: 138702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4612.218957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.210929] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4617.709465] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.819405] LustreError: 138702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.177554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.520985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4632.777580] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4641.403722] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4647.844574] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4654.990333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.469947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.477752] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4671.137250] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4676.511143] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4676.859607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4682.172517] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4682.415785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4691.473716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4691.670418] Lustre: lustre-OST0001: new disk, initializing [ 4691.673222] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4691.677797] Lustre: 142075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4691.751931] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4697.298589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4697.309605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4697.456106] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4698.604575] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4707.993499] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4708.222887] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4712.562712] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4712.800217] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4719.341762] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90d891df5000.ost_server_uuid 50 [ 4721.560385] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90d891df5000.ost_server_uuid in FULL state after 0 sec [ 4733.411326] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4733.413020] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.419982] LustreError: Skipped 1 previous similar message [ 4733.425340] Lustre: Skipped 1 previous similar message [ 4733.426632] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4733.426640] Lustre: Skipped 2 previous similar messages [ 4734.875906] Lustre: server umount lustre-OST0001 complete [ 4738.550066] LustreError: 141727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.574994] LustreError: 141727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4743.651505] LustreError: 140524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4743.653279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.654323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.654330] Lustre: Skipped 1 previous similar message [ 4743.684467] LustreError: 140524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4743.717390] Lustre: Skipped 2 previous similar messages [ 4745.856652] Lustre: server umount lustre-OST0000 complete [ 4750.305200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4750.313698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4750.329323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4750.334255] Lustre: Skipped 1 previous similar message [ 4756.259968] Lustre: server umount lustre-MDT0000 complete [ 4759.015913] LustreError: 143148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4759.041874] LustreError: 143148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4759.947157] LustreError: 142073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442724 with bad export cookie 13529763791533487530 [ 4759.959737] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4759.960561] LustreError: 142073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4760.337812] Lustre: server umount lustre-MDT0001 complete [ 4784.476928] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 4788.090477] Key type lgssc unregistered [ 4788.361983] LNet: 145078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4788.369044] LNetError: 145078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4788.379858] LNet: Removed LNI 192.168.202.140@tcp [ 4789.359444] Key type .llcrypt unregistered [ 4789.363339] Key type ._llcrypt unregistered [ 4833.843629] Key type ._llcrypt registered [ 4833.846931] Key type .llcrypt registered [ 4833.996295] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 4849.154221] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 4850.759723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4850.804168] alg: No test for adler32 (adler32-zlib) [ 4851.859823] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4852.139437] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 4853.840521] Key type lgssc registered [ 4855.022661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4892.381542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4892.452809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.835634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4893.887206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4893.991989] Lustre: lustre-MDT0000: new disk, initializing [ 4894.102100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.122960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4898.861207] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4909.932585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4910.038820] Lustre: 149233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4910.093817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4910.097439] Lustre: Skipped 1 previous similar message [ 4910.167997] Lustre: lustre-MDT0001: new disk, initializing [ 4910.296044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4910.332480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4910.351075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4916.055771] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4926.507446] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4933.078364] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4939.727364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.125662] Lustre: lustre-OST0000: new disk, initializing [ 4940.132566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4940.143164] Lustre: 151202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4940.284867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.473735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4945.486163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4945.632239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4946.973994] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4958.076566] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4964.182520] Lustre: Failing over lustre-OST0000 [ 4964.631030] Lustre: server umount lustre-OST0000 complete [ 4981.730072] Lustre: 147457:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442929/real 1788442929] req@00000000fe3aa861 x1875318214264960/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788442945 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4981.770723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.747735] Lustre: 147456:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442930/real 1788442930] req@00000000d12d2277 x1875318214266112/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442946 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4982.793555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4983.843155] Lustre: 147455:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442930/real 1788442930] req@00000000ae6d3334 x1875318214265728/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4983.874841] Lustre: 147455:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4984.837616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4985.167302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4985.193411] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4987.170339] Lustre: 147456:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442935/real 1788442935] req@00000000103a615e x1875318214267264/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442951 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4987.209554] Lustre: 147456:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4991.351619] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4992.069361] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 4992.289408] Lustre: 147455:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442940/real 1788442940] req@000000002172546b x1875318214268288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4992.931320] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4992.931485] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4996.178181] Lustre: Failing over lustre-MDT0000 [ 4996.438211] Lustre: server umount lustre-MDT0000 complete [ 4996.461711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4996.476249] Lustre: Skipped 2 previous similar messages [ 5001.698848] LustreError: 150397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.738473] LustreError: 150397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5003.071967] LustreError: 149241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.817418] LustreError: 149245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5006.839974] LustreError: 149245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5011.939138] LustreError: 149244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5011.971723] LustreError: 149244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5015.716950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.835230] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5016.114940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.172588] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5018.449590] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5021.169263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5021.171951] Lustre: Skipped 1 previous similar message [ 5021.184696] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5021.254374] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5021.299323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5031.395640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5031.400189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5031.433913] Lustre: Skipped 1 previous similar message [ 5031.452946] Lustre: Skipped 1 previous similar message [ 5036.515282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5036.522043] Lustre: Skipped 1 previous similar message [ 5036.913289] Lustre: server umount lustre-OST0000 complete [ 5041.642247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.647667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5041.650728] Lustre: Skipped 1 previous similar message [ 5041.656958] Lustre: Skipped 1 previous similar message [ 5046.754608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5046.765014] Lustre: Skipped 1 previous similar message [ 5047.487319] Lustre: server umount lustre-MDT0000 complete [ 5051.541994] LustreError: 149225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443015 with bad export cookie 14638202436125985201 [ 5051.554157] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.558992] LustreError: 149225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5051.996875] Lustre: server umount lustre-MDT0001 complete [ 5067.832288] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 5071.694776] Key type lgssc unregistered [ 5072.109779] LNet: 154858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5072.121190] LNetError: 154858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5072.146444] LNet: Removed LNI 192.168.202.140@tcp [ 5073.215899] Key type .llcrypt unregistered [ 5073.218114] Key type ._llcrypt unregistered [ 5113.996995] Key type ._llcrypt registered [ 5113.999842] Key type .llcrypt registered [ 5114.124566] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5115.146358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5115.217416] alg: No test for adler32 (adler32-zlib) [ 5116.501513] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5116.885165] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 5118.688179] Key type lgssc registered [ 5120.551640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5131.431686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5131.468372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5132.951651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.344650] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5142.511530] LustreError: 156379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.544083] LustreError: 156379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5147.609621] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5147.624790] LustreError: 156380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5151.610631] LustreError: 156408:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5151.618480] LustreError: 156408:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5151.621583] LustreError: 156408:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 5151.896798] Lustre: server umount lustre-MDT0000 complete [ 5168.274501] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 5180.979736] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5227.780846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.224932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5228.317884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5228.557638] Lustre: lustre-MDT0000: new disk, initializing [ 5228.785234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5228.838243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5236.161211] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5248.086631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.168498] Lustre: 159717:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5248.208041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5248.211099] Lustre: Skipped 1 previous similar message [ 5248.293592] Lustre: lustre-MDT0001: new disk, initializing [ 5248.361989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5248.382779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5248.391816] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5252.582964] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5264.280821] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5271.421963] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5278.804199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.177350] Lustre: lustre-OST0000: new disk, initializing [ 5279.182753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5279.192315] Lustre: 161683:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5279.365437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5287.297852] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5288.028456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5288.111239] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5288.211090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5297.860188] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5302.328187] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5304.604654] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:47:46 (1788443266) [ 5585.376805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.378418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5585.402042] Lustre: Skipped 1 previous similar message [ 5585.420779] Lustre: Skipped 2 previous similar messages [ 5590.497936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5590.503944] Lustre: Skipped 2 previous similar messages [ 5590.978562] Lustre: server umount lustre-MDT0000 complete [ 5595.170660] LustreError: 161682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443559 with bad export cookie 17848654014902591937 [ 5595.173505] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5595.179641] LustreError: 161682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5595.619257] LustreError: 159723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.620660] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.639824] Lustre: Skipped 1 previous similar message [ 5595.642749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5600.736871] LustreError: 162065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5600.737226] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5601.685396] Lustre: server umount lustre-MDT0001 complete [ 5612.648653] Lustre: server umount lustre-OST0000 complete [ 5625.404441] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 5636.797668] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5681.326060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.461097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5681.478088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5681.536434] Lustre: lustre-MDT0000: new disk, initializing [ 5681.608942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.632530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5686.686143] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5697.265294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.437461] Lustre: 166364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5697.544809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5697.551870] Lustre: Skipped 1 previous similar message [ 5697.649194] Lustre: lustre-MDT0001: new disk, initializing [ 5697.771673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.829291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5697.842613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5702.208042] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5712.316377] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5717.981959] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5725.693879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5725.960844] Lustre: lustre-OST0000: new disk, initializing [ 5725.965737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5725.989761] Lustre: 168331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5726.125486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5732.239694] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5733.444427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5733.461465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5733.598379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5741.631495] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5746.136646] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5746.352876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5752.625275] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5752.902700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5759.457405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5759.464423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.474584] Lustre: Skipped 1 previous similar message [ 5759.475725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5759.499966] Lustre: Skipped 1 previous similar message [ 5761.563585] Lustre: server umount lustre-OST0000 complete [ 5769.702837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.706498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5769.711820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.728353] Lustre: Skipped 1 previous similar message [ 5769.743557] LustreError: Skipped 1 previous similar message [ 5774.556771] Lustre: server umount lustre-MDT0000 complete [ 5774.842351] LustreError: 167974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.898560] LustreError: 167974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5780.729108] LustreError: 166356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443744 with bad export cookie 17848654014902750767 [ 5780.730208] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5780.734826] LustreError: 166356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.108410] Lustre: server umount lustre-MDT0001 complete [ 5794.851712] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:55:56 (1788443756) [ 5816.333894] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 5826.875140] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 5868.006538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5868.390744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5868.413258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5868.512461] Lustre: lustre-MDT0000: new disk, initializing [ 5868.650544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.682524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5874.366555] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5887.229934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.358255] Lustre: 173088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5887.696845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5894.702183] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5905.539258] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5912.459910] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5919.825495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5920.185460] Lustre: lustre-OST0000: new disk, initializing [ 5920.195789] Lustre: Skipped 1 previous similar message [ 5920.207323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5920.212782] Lustre: Skipped 2 previous similar messages [ 5920.220926] Lustre: 175067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5927.183141] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 5929.081822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5929.084161] Lustre: Skipped 1 previous similar message [ 5929.102732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5929.256581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5937.637845] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5949.921503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.934771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.937855] Lustre: Skipped 1 previous similar message [ 5949.953416] Lustre: Skipped 2 previous similar messages [ 5952.064496] Lustre: server umount lustre-OST0000 complete [ 5959.649471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5959.654773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.504662] Lustre: server umount lustre-MDT0000 complete [ 5965.284772] LustreError: 174288:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.318797] LustreError: 174288:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5968.044264] LustreError: 176172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443932 with bad export cookie 17848654014902751649 [ 5968.053199] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.062578] LustreError: 176172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5968.696377] Lustre: server umount lustre-MDT0001 complete [ 5979.657623] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:59:01 (1788443941) [ 6000.339932] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 6011.596607] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6051.932209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6052.203273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6052.240265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6052.331938] Lustre: lustre-MDT0000: new disk, initializing [ 6052.440851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6052.451544] Lustre: Skipped 2 previous similar messages [ 6052.466615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6057.680955] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6068.779390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.921746] Lustre: 179394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6069.229596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6074.183517] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6084.839765] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6092.840294] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6101.168746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6101.619404] Lustre: 181385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6109.857110] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6110.362384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6110.518209] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6121.519836] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6131.169288] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6131.183140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6131.200269] Lustre: Skipped 2 previous similar messages [ 6131.210283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6131.213975] Lustre: Skipped 3 previous similar messages [ 6136.296656] LustreError: 181360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.327656] LustreError: 181360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6136.764970] Lustre: server umount lustre-OST0000 complete [ 6146.016928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6146.019645] LustreError: Skipped 1 previous similar message [ 6147.900389] Lustre: server umount lustre-MDT0000 complete [ 6152.410614] LustreError: 181384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444116 with bad export cookie 17848654014902752734 [ 6152.413352] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6152.431880] LustreError: 181384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6153.359740] Lustre: server umount lustre-MDT0001 complete [ 6166.540976] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6170.451325] Key type lgssc unregistered [ 6170.813369] LNet: 183240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6170.826515] LNetError: 183240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6170.855623] LNet: Removed LNI 192.168.202.140@tcp [ 6171.884785] Key type .llcrypt unregistered [ 6171.886839] Key type ._llcrypt unregistered [ 6195.227418] Key type ._llcrypt registered [ 6195.231274] Key type .llcrypt registered [ 6195.282209] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6196.276679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6196.499972] alg: No test for adler32 (adler32-zlib) [ 6197.751766] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6198.030469] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6199.722139] Key type lgssc registered [ 6200.619381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.141225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.188285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.113645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6214.649734] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6220.257776] LustreError: 184654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.281689] LustreError: 184654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6223.815071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6224.351994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6229.453490] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6237.555788] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6244.282669] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6251.743182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.316918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6255.417191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6259.717844] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6268.206416] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6290.404480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.405020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6290.421272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6293.368577] Lustre: server umount lustre-OST0000 complete [ 6300.641989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6300.649474] Lustre: Skipped 1 previous similar message [ 6300.658607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6300.661170] Lustre: Skipped 1 previous similar message [ 6303.742385] Lustre: server umount lustre-MDT0000 complete [ 6305.764424] LustreError: 184654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.773526] LustreError: 184654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6306.696355] LustreError: 184634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444270 with bad export cookie 17142704853819107530 [ 6306.699069] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6306.705073] LustreError: 184634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6307.025474] Lustre: server umount lustre-MDT0001 complete [ 6321.442226] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6325.067294] Key type lgssc unregistered [ 6325.547160] LNet: 188212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6325.572968] LNetError: 188212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6325.592052] LNet: Removed LNI 192.168.202.140@tcp [ 6326.837962] Key type .llcrypt unregistered [ 6326.840526] Key type ._llcrypt unregistered [ 6351.255458] Key type ._llcrypt registered [ 6351.257895] Key type .llcrypt registered [ 6351.386434] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6352.440809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6352.517128] alg: No test for adler32 (adler32-zlib) [ 6353.590434] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6353.929382] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 6355.592401] Key type lgssc registered [ 6356.759382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6364.586921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.601259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.058170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6371.510631] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6376.420480] LustreError: 189626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6376.455271] LustreError: 189626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6381.541599] LustreError: 189627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6382.476103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.896037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6388.715064] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6397.133634] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6403.779315] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6413.432958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6413.908955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6422.036905] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6431.162607] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.482265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6442.977516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.980612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.005102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6447.075353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6447.076475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6447.097085] Lustre: Skipped 1 previous similar message [ 6448.447196] Lustre: lustre-OST0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 6448.449490] Lustre: Skipped 1 previous similar message [ 6448.776351] Lustre: server umount lustre-OST0000 complete [ 6457.710842] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90d8986af000.ost_server_uuid 50 [ 6459.383553] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6475.234752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6475.239670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.251052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.798440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.439809] Lustre: server umount lustre-MDT0000 complete [ 6482.925566] LustreError: 189627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.961768] LustreError: 189627:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6483.192064] LustreError: 189607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444447 with bad export cookie 8123886344874329139 [ 6483.195478] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.208261] LustreError: 189607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6483.654845] Lustre: server umount lustre-MDT0001 complete [ 6492.498172] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:07:34 (1788444454) [ 6501.151508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.850416] LustreError: 193544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.026448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.489067] LustreError: 193545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6508.844026] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6516.707713] LustreError: 193544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.736770] LustreError: 193544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6519.300510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.733524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6524.871199] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6532.952963] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6539.349667] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6547.164694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.369945] LustreError: 195367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.588703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.087554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6554.831661] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6562.903799] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.090577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6570.287696] Lustre: lustre-OST0001: new disk, initializing [ 6570.292439] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6570.297474] Lustre: 196389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6570.402620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6576.938409] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6580.364099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6580.378839] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6580.533947] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6587.190050] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.948921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.954409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6590.961628] Lustre: Skipped 1 previous similar message [ 6590.965534] Lustre: Skipped 2 previous similar messages [ 6596.852748] Lustre: server umount lustre-OST0000 complete [ 6597.951396] LustreError: 196085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.061829] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6604.204685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6609.998334] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6610.412643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6626.786976] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.798603] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6626.817545] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6626.837017] Lustre: Skipped 5 previous similar messages [ 6632.241914] Lustre: server umount lustre-OST0001 complete [ 6642.410372] Lustre: server umount lustre-MDT0000 complete [ 6646.483587] LustreError: 193525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444610 with bad export cookie 8123886344874330217 [ 6646.486153] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.499454] LustreError: 193525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6647.004726] Lustre: server umount lustre-MDT0001 complete [ 6672.796243] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:10:35 (1788444635) [ 6679.796668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6680.148953] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6680.194062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6680.353476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.358833] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6694.724764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.843616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6694.894516] Lustre: 199831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6699.865970] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6707.787742] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6713.983641] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6720.955326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6721.186409] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6729.235366] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6730.516021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6739.932741] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6749.542270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.597103] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6749.981902] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6749.990347] Lustre: Skipped 2 previous similar messages [ 6756.150556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6756.733829] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6764.116475] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.564044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6766.574896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.577634] Lustre: Skipped 3 previous similar messages [ 6766.588810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6766.593120] Lustre: Skipped 7 previous similar messages [ 6772.648387] Lustre: server umount lustre-OST0000 complete [ 6775.120379] LustreError: 201682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.133706] LustreError: 201682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 6792.179046] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.199614] LustreError: Skipped 1 previous similar message [ 6794.183846] Lustre: server umount lustre-OST0001 complete [ 6802.402527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.407596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6802.421758] Lustre: Skipped 3 previous similar messages [ 6802.430995] Lustre: Skipped 6 previous similar messages [ 6805.827252] Lustre: server umount lustre-MDT0000 complete [ 6810.516327] LustreError: 199822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444774 with bad export cookie 8123886344874331624 [ 6810.519828] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6810.526917] LustreError: 199822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6811.247142] Lustre: server umount lustre-MDT0001 complete [ 6838.629828] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:13:21 (1788444801) [ 6855.908978] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 6866.370409] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 6907.762786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.096568] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6908.126804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6908.239991] Lustre: lustre-MDT0000: new disk, initializing [ 6908.344615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.369270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6912.938630] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6924.213531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.396936] Lustre: 207925:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6924.454203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6924.461975] Lustre: Skipped 1 previous similar message [ 6924.591941] Lustre: lustre-MDT0001: new disk, initializing [ 6924.722201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6924.739828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6929.897245] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6940.115099] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6945.752683] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6951.753566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6952.236183] Lustre: lustre-OST0000: new disk, initializing [ 6952.242118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6952.254217] Lustre: 209893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6959.664794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6959.685440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6959.850948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6960.329959] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 6972.237296] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6979.688684] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6980.342220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6987.068909] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6987.579897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6990.821991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6990.826497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.826947] LustreError: Skipped 2 previous similar messages [ 6990.828955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.878887] Lustre: Skipped 1 previous similar message [ 6995.609428] Lustre: server umount lustre-OST0000 complete [ 7014.368117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7014.532308] Lustre: server umount lustre-MDT0000 complete [ 7016.432284] LustreError: 207937:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7016.445319] LustreError: 207937:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7017.936736] LustreError: 207918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444981 with bad export cookie 8123886344874333409 [ 7017.938807] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7017.947601] LustreError: 207918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7018.549815] Lustre: server umount lustre-MDT0001 complete [ 7028.038574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7035.654606] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7046.515354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7052.092710] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7062.143601] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7069.572454] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.397188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.419844] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7094.751705] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7100.252525] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7100.466684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7106.079191] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7106.292730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7115.015887] Lustre: server umount lustre-OST0000 complete [ 7121.330395] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7125.910769] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7155.922452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7163.671465] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7164.654119] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7164.662732] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7164.665412] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7173.147901] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7174.949381] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7185.382548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.383404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.394395] LustreError: Skipped 1 previous similar message [ 7185.395091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7185.395096] Lustre: Skipped 11 previous similar messages [ 7185.458768] Lustre: Skipped 5 previous similar messages [ 7188.666217] Lustre: server umount lustre-OST0000 complete [ 7200.012303] Lustre: server umount lustre-MDT0000 complete [ 7204.311552] LustreError: 212096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445168 with bad export cookie 8123886344874334291 [ 7204.312749] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7204.322668] LustreError: 212096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7204.745524] Lustre: server umount lustre-MDT0001 complete [ 7215.241510] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:19:37 (1788445177) [ 7222.787764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.403274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7223.412707] Lustre: Skipped 6 previous similar messages [ 7228.576159] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7238.222480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.299877] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7251.851649] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7258.389774] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7265.214435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7271.042089] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7279.468852] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7286.871152] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7287.173791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7294.736799] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7295.198862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7302.880860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.096293] Lustre: lustre-OST0001: new disk, initializing [ 7303.099710] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7303.107915] Lustre: 221457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7308.879818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7308.891198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7309.050264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7310.722917] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7321.202274] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7326.834859] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7327.107398] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7332.880811] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7333.056440] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7340.023801] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7340.032699] LustreError: Skipped 2 previous similar messages [ 7341.656715] Lustre: server umount lustre-OST0001 complete [ 7345.130101] LustreError: 219906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7345.155704] LustreError: 219906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7348.225472] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7348.555711] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7354.593489] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7354.878098] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7385.390765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7390.737453] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7390.753784] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7390.753900] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7390.762335] Lustre: Skipped 1 previous similar message [ 7392.508561] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7402.138334] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.595980] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7411.186335] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7413.189836] Lustre: server umount lustre-OST0001 complete [ 7437.381778] Lustre: server umount lustre-OST0000 complete [ 7441.894496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.895791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.910309] Lustre: Skipped 9 previous similar messages [ 7441.923856] Lustre: Skipped 12 previous similar messages [ 7447.876121] Lustre: server umount lustre-MDT0000 complete [ 7452.057382] LustreError: 218066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445416 with bad export cookie 8123886344874335257 [ 7452.072154] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.077790] LustreError: 218066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7458.413268] Lustre: server umount lustre-MDT0001 complete [ 7492.608690] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:24:14 (1788445454) [ 7501.935274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7502.399989] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7502.460949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7508.064831] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7518.620988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7518.798560] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7518.834891] Lustre: 226703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7525.125557] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7534.099863] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7542.031753] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7550.590954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7550.811731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7558.970657] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7568.735501] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7579.695742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7579.796764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7586.778784] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7595.011189] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7597.384341] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7602.804749] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7603.095456] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7608.575316] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7608.806253] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7615.848397] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90d888323800.ost_server_uuid 50 [ 7617.612993] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90d888323800.ost_server_uuid in IDLE state after 0 sec [ 7618.812105] Lustre: Permanently deactivating lustre-OST0001 [ 7618.825724] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7621.958942] Lustre: Permanently reactivating lustre-OST0001 [ 7628.269866] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7628.278603] LustreError: Skipped 2 previous similar messages [ 7632.617146] Lustre: server umount lustre-OST0001 complete [ 7652.322756] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7652.749428] Lustre: server umount lustre-OST0000 complete [ 7671.264266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7671.577693] Lustre: server umount lustre-MDT0000 complete [ 7676.413482] LustreError: 229591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445640 with bad export cookie 8123886344874336545 [ 7676.415527] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.429023] LustreError: 229591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7676.949435] Lustre: server umount lustre-MDT0001 complete [ 7705.376479] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:27:47 (1788445667) [ 7714.937298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7715.162941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7715.196223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7715.207235] Lustre: Skipped 5 previous similar messages [ 7720.555679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7730.873230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.053035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7731.113749] Lustre: 233410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7736.643285] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7745.929592] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7752.727911] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7760.806625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7761.058696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7761.094570] Lustre: Permanently deactivating lustre-OST0000 [ 7761.108808] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7761.289702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.295251] Lustre: Skipped 10 previous similar messages [ 7769.325311] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7771.199234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7778.544920] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7786.571750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7786.681980] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7794.657438] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7795.383879] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7802.651961] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.705499] Lustre: Permanently reactivating lustre-OST0000 [ 7804.719291] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7804.729977] Lustre: Skipped 2 previous similar messages [ 7813.666152] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7813.673869] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000c2977438, cur 1788445777 deadline 1788445765 last 1788445735 [ 7813.726350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7821.400460] Lustre: server umount lustre-OST0001 complete [ 7832.733985] Lustre: server umount lustre-OST0000 complete [ 7843.307501] Lustre: server umount lustre-MDT0000 complete [ 7849.161631] LustreError: 233402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445813 with bad export cookie 8123886344874338225 [ 7849.168843] LustreError: 233402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7849.169510] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7850.051310] Lustre: server umount lustre-MDT0001 complete [ 7864.563875] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 7867.246385] Key type lgssc unregistered [ 7867.639885] LNet: 238293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7867.644760] LNetError: 238293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7868.709936] LNet: Removed LNI 192.168.202.140@tcp [ 7869.615756] Key type .llcrypt unregistered [ 7869.621672] Key type ._llcrypt unregistered [ 7895.733907] Key type ._llcrypt registered [ 7895.739232] Key type .llcrypt registered [ 7895.834755] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 7896.767800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7896.898835] alg: No test for adler32 (adler32-zlib) [ 7898.006231] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7898.298296] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 7900.008180] Key type lgssc registered [ 7901.430597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7911.771163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7911.789650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7913.348390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.413649] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7923.687924] LustreError: 239760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.720252] LustreError: 239760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7928.074960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.274711] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7928.323180] Lustre: 239748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7928.346296] Lustre: Permanently deactivating lustre-MDT0001 [ 7928.361822] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7928.713069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7934.002994] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7934.194995] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7942.278595] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7948.436937] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7956.500686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.765395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.848993] LustreError: 241600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.902366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7963.738948] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7964.138305] LustreError: 241599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7969.262563] LustreError: 241600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.354052] LustreError: 241600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.442672] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7973.578686] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7981.583286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7981.968254] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.207118] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7987.213726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7988.957842] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 7996.943518] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7999.720319] Lustre: Permanently reactivating lustre-MDT0001 [ 7999.733732] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7999.737822] Lustre: Skipped 2 previous similar messages [ 8004.976476] Lustre: lustre-MDT0001: Client 751c72a0-f969-419c-9a05-abd32cedd85b (at 192.168.202.40@tcp) reconnecting [ 8008.850414] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.884720] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 8008.894926] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 8008.905593] LustreError: 243464:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 8008.918686] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8008.929664] LustreError: 243464:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 8008.929782] LustreError: 243464:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.140@tcp: cfg command failed: rc = -108 [ 8008.929939] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 8008.929939] [ 8008.932924] LustreError: 239752:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 8010.833491] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8012.385252] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8017.455219] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8017.622361] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8022.472633] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8022.862509] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8027.403046] Lustre: Permanently deactivating lustre-MDT0001 [ 8027.427347] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 8027.435269] Lustre: Skipped 2 previous similar messages [ 8036.505163] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8044.515553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.518372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8044.538051] Lustre: Skipped 1 previous similar message [ 8044.548413] Lustre: Skipped 3 previous similar messages [ 8049.637138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.646238] Lustre: Skipped 3 previous similar messages [ 8050.424721] Lustre: server umount lustre-MDT0000 complete [ 8054.164546] LustreError: 239741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446018 with bad export cookie 18306292857229523164 [ 8054.174498] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8054.180306] LustreError: 239741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8054.772683] Lustre: server umount lustre-MDT0001 complete [ 8061.394262] Lustre: server umount lustre-OST0000 complete [ 8072.616877] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:33:55 (1788446035) [ 8083.905958] Lustre: server umount lustre-OST0001 complete [ 8090.792872] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 8101.204220] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 8143.817227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.103926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8144.110836] Lustre: Skipped 2 previous similar messages [ 8144.167898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8144.273804] Lustre: lustre-MDT0000: new disk, initializing [ 8144.428413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8144.449299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8149.954260] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8162.023775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.128236] Lustre: 248020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8162.181059] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8162.187773] Lustre: Skipped 1 previous similar message [ 8162.256534] Lustre: lustre-MDT0001: new disk, initializing [ 8162.373827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8162.437861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8162.442230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8167.547693] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8178.209845] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8185.544932] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8193.640340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.063932] Lustre: lustre-OST0000: new disk, initializing [ 8194.075637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8194.087443] Lustre: 249992:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8194.243228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.079451] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8204.366279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8204.400978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8204.515312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8211.064604] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8213.283585] LustreError: 250277:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8222.698809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8222.916437] Lustre: lustre-OST0001: new disk, initializing [ 8222.924214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8222.929695] Lustre: 251107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8223.344382] LustreError: 250277:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8223.363423] LustreError: 248027:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8230.195621] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8232.582189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8232.598957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8232.742563] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8233.369901] LustreError: 248027:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8240.402896] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8242.230419] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8248.303842] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8248.309556] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8248.310197] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8248.317841] Lustre: Skipped 3 previous similar messages [ 8253.417120] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8253.426826] Lustre: Skipped 2 previous similar messages [ 8258.536654] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8258.552167] Lustre: Skipped 1 previous similar message [ 8259.040800] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8259.168926] Lustre: server umount lustre-OST0001 complete [ 8263.658257] LustreError: 250474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8263.686935] LustreError: 250474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8276.454127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8276.466134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.481481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8276.483994] Lustre: Skipped 1 previous similar message [ 8276.500986] Lustre: Skipped 1 previous similar message [ 8284.147343] LustreError: 249973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.165612] LustreError: 249973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8290.272130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8290.566982] Lustre: server umount lustre-OST0000 complete [ 8295.393487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8295.404272] LustreError: Skipped 1 previous similar message [ 8295.406166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.424947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8295.438873] Lustre: Skipped 6 previous similar messages [ 8300.737099] Lustre: server umount lustre-MDT0000 complete [ 8304.516981] LustreError: 248814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446268 with bad export cookie 18306292857229526202 [ 8304.522715] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.529835] LustreError: 248814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8311.156082] Lustre: server umount lustre-MDT0001 complete [ 8324.558511] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 8328.879372] Key type lgssc unregistered [ 8329.248821] LNet: 253144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8329.255209] LNetError: 253144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8330.281755] LNet: Removed LNI 192.168.202.140@tcp [ 8331.311475] Key type .llcrypt unregistered [ 8331.313756] Key type ._llcrypt unregistered [ 8377.393138] Key type ._llcrypt registered [ 8377.397297] Key type .llcrypt registered [ 8377.513741] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 8378.485222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8378.596753] alg: No test for adler32 (adler32-zlib) [ 8380.014380] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8380.275955] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 8381.969993] Key type lgssc registered [ 8383.812484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8393.643058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8393.659920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.061725] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8395.107789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8395.406797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.873880] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8409.807440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.931891] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8409.967636] Lustre: 255352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8410.256990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8416.151325] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8425.771456] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8432.145177] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8439.928451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8440.229836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8440.458951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8447.213780] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8449.134633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8455.892241] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.951609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.001183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8515.554471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8515.562242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8515.592613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8517.037809] Lustre: server umount lustre-OST0000 complete [ 8521.431661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8534.068125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8534.456025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8541.619354] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8541.635836] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8541.642555] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8541.677776] Lustre: Skipped 1 previous similar message [ 8542.070673] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8551.005781] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8592.867128] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8592.887797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8592.901022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8592.908323] Lustre: Skipped 3 previous similar messages [ 8598.014895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8598.027950] Lustre: Skipped 1 previous similar message [ 8598.314184] Lustre: server umount lustre-OST0000 complete [ 8604.643107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8604.646202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8604.669293] Lustre: Skipped 1 previous similar message [ 8604.680851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8609.452567] Lustre: server umount lustre-MDT0000 complete [ 8613.519195] LustreError: 255342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446577 with bad export cookie 6040486072489100379 [ 8613.520953] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8613.526843] LustreError: 255342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8613.956751] Lustre: server umount lustre-MDT0001 complete [ 8627.107873] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 8630.813973] Key type lgssc unregistered [ 8631.065803] LNet: 260774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8631.071357] LNetError: 260774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8631.089768] LNet: Removed LNI 192.168.202.140@tcp [ 8632.051239] Key type .llcrypt unregistered [ 8632.052655] Key type ._llcrypt unregistered [ 8658.502080] Key type ._llcrypt registered [ 8658.504623] Key type .llcrypt registered [ 8658.740666] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 8659.934905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8660.050051] alg: No test for adler32 (adler32-zlib) [ 8661.199151] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8661.397647] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 8663.040284] Key type lgssc registered [ 8664.446641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8672.339067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8672.362495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8673.783208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.309941] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8682.995107] LustreError: 262199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8683.028199] LustreError: 262199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8687.885044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.210853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8692.904460] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8702.582528] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8710.713655] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8718.850895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8719.405406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8726.643619] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8734.819774] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8742.964369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8773.603810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8773.615701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8773.667455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8774.625241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8774.645509] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8774.676882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.678499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.684595] Lustre: Skipped 1 previous similar message [ 8778.087813] Lustre: server umount lustre-OST0000 complete [ 8785.383534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8785.386785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8785.406369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8788.912903] Lustre: server umount lustre-MDT0000 complete [ 8792.034820] LustreError: 262196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.052820] LustreError: 262196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8792.884381] LustreError: 262180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446756 with bad export cookie 9812496100038150620 [ 8792.888152] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8792.898227] LustreError: 262180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8793.404407] Lustre: server umount lustre-MDT0001 complete [ 8804.474955] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 8807.385450] Key type lgssc unregistered [ 8807.653734] LNet: 266530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8807.667679] LNetError: 266530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8808.761587] LNet: Removed LNI 192.168.202.140@tcp [ 8809.842660] Key type .llcrypt unregistered [ 8809.848339] Key type ._llcrypt unregistered [ 8827.535396] Key type ._llcrypt registered [ 8827.537745] Key type .llcrypt registered [ 8827.644760] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 8828.517273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8828.548249] alg: No test for adler32 (adler32-zlib) [ 8829.675928] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8829.915436] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 8831.624449] Key type lgssc registered [ 8833.043651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8841.049803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8841.083720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.632092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8847.421595] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8851.954360] LustreError: 267751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8851.973350] LustreError: 267751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8857.059349] LustreError: 267752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.194451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.444725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8863.242715] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8871.805257] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8880.038572] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8889.334071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.731855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8896.886227] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 8905.941608] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8912.314199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8922.596528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8922.601214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8922.614319] Lustre: Skipped 1 previous similar message [ 8922.618129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8922.625229] Lustre: Skipped 1 previous similar message [ 8927.716141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8927.723404] Lustre: Skipped 1 previous similar message [ 8928.922854] Lustre: server umount lustre-OST0000 complete [ 8932.833928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.857905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.864349] Lustre: Skipped 1 previous similar message [ 8937.956553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8937.973031] Lustre: Skipped 1 previous similar message [ 8943.078195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8943.087546] Lustre: Skipped 1 previous similar message [ 8947.169186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8947.479310] Lustre: server umount lustre-MDT0000 complete [ 8948.196508] LustreError: 269187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.225101] LustreError: 269187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8951.637778] LustreError: 267731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446915 with bad export cookie 1257953344053624859 [ 8951.653752] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8952.092993] Lustre: server umount lustre-MDT0001 complete [ 8963.842138] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 8967.076854] Key type lgssc unregistered [ 8967.410591] LNet: 271433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8967.421142] LNetError: 271433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8967.446497] LNet: Removed LNI 192.168.202.140@tcp [ 8968.554712] Key type .llcrypt unregistered [ 8968.558102] Key type ._llcrypt unregistered [ 8992.970452] Key type ._llcrypt registered [ 8992.972795] Key type .llcrypt registered [ 8993.102614] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 8994.021815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8994.049262] alg: No test for adler32 (adler32-zlib) [ 8995.413795] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8995.734910] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 8997.456793] Key type lgssc registered [ 8998.825339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9007.593750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9007.623345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.981250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.922654] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9019.368025] LustreError: 272845:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.394819] LustreError: 272845:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9023.024216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.447032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9028.084438] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9035.442582] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9040.752403] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9047.654200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9047.981574] LustreError: 274667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9048.108823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9054.227503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9054.683036] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9063.107081] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9090.018892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9090.024377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9090.042505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9095.141699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9095.149555] Lustre: Skipped 1 previous similar message [ 9095.735834] Lustre: server umount lustre-OST0000 complete [ 9105.382360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.398996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9105.401884] Lustre: Skipped 2 previous similar messages [ 9105.417223] Lustre: Skipped 2 previous similar messages [ 9107.055947] Lustre: server umount lustre-MDT0000 complete [ 9110.508128] LustreError: 272846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.526110] LustreError: 272846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9111.332896] LustreError: 272825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447075 with bad export cookie 7032823494452000810 [ 9111.348935] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9111.826830] Lustre: server umount lustre-MDT0001 complete [ 9124.731915] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 9128.306845] Key type lgssc unregistered [ 9128.833954] LNet: 277150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9128.841219] LNetError: 277150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9128.861058] LNet: Removed LNI 192.168.202.140@tcp [ 9130.186410] Key type .llcrypt unregistered [ 9130.189402] Key type ._llcrypt unregistered [ 9151.226254] Key type ._llcrypt registered [ 9151.228812] Key type .llcrypt registered [ 9151.390296] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 9152.992622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9153.411461] alg: No test for adler32 (adler32-zlib) [ 9154.505721] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9154.915162] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 9156.656795] Key type lgssc registered [ 9157.967461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9166.997483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9167.011911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.417200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9173.413414] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9178.605508] LustreError: 278388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9178.616352] LustreError: 278388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9179.619293] LustreError: 278390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9182.152702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.663244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9187.697043] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9196.161293] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9202.435876] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9210.683462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.115510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.208449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9218.575184] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9226.249689] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9244.129987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9244.155344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.172416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9244.185644] Lustre: Skipped 1 previous similar message [ 9245.992523] Lustre: server umount lustre-OST0000 complete [ 9254.375665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9254.378591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.379595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9254.379601] Lustre: Skipped 1 previous similar message [ 9254.426269] Lustre: Skipped 1 previous similar message [ 9256.515437] Lustre: server umount lustre-MDT0000 complete [ 9259.493947] LustreError: 278399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.510901] LustreError: 278399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9260.880744] LustreError: 278354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447224 with bad export cookie 9062814028277347853 [ 9260.897811] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9261.629906] Lustre: server umount lustre-MDT0001 complete [ 9272.647486] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 9275.822919] Key type lgssc unregistered [ 9276.179050] LNet: 282088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9276.191231] LNetError: 282088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9276.212692] LNet: Removed LNI 192.168.202.140@tcp [ 9277.149495] Key type .llcrypt unregistered [ 9277.157643] Key type ._llcrypt unregistered [ 9289.115316] Key type ._llcrypt registered [ 9289.117826] Key type .llcrypt registered [ 9289.219332] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9307.499782] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 9323.764475] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 9324.986265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9325.013947] alg: No test for adler32 (adler32-zlib) [ 9326.089342] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9326.343493] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 9328.000209] Key type lgssc registered [ 9329.157806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9364.297676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.352903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.669820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9365.737994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9365.880721] Lustre: lustre-MDT0000: new disk, initializing [ 9366.018505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9366.068826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9369.889284] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9380.290220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9380.441290] Lustre: 285634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9380.516525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9380.522352] Lustre: Skipped 1 previous similar message [ 9380.653944] Lustre: lustre-MDT0001: new disk, initializing [ 9380.783505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9380.833077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9380.857280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9386.035548] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9398.298686] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9404.699100] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9411.410204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9411.908503] Lustre: lustre-OST0000: new disk, initializing [ 9411.912701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9411.921680] Lustre: 287604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9412.049378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.361139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9419.381007] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9419.554041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9421.097723] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9433.643520] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9438.181437] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9438.488729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9443.702270] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9443.971419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9450.469845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9450.476537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9450.484770] LustreError: Skipped 1 previous similar message [ 9450.500168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9452.786184] Lustre: server umount lustre-OST0000 complete [ 9460.706195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9460.729488] Lustre: Skipped 1 previous similar message [ 9460.733904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9460.740099] Lustre: Skipped 2 previous similar messages [ 9464.260079] Lustre: server umount lustre-MDT0000 complete [ 9468.008922] LustreError: 285628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447432 with bad export cookie 3143651517888725936 [ 9468.015146] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9468.029035] LustreError: 285628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9468.369565] Lustre: server umount lustre-MDT0001 complete [ 9477.459509] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:57:19 (1788447439) [ 9483.983317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.485290] LustreError: 290018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.514120] LustreError: 290018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9484.610261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.124258] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9489.894829] LustreError: 290019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9493.990227] LustreError: 290018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.749219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.168187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.440876] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9512.675958] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9519.261288] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9526.266310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9526.720692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9534.935278] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9543.066779] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9553.379493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.393877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9553.394735] Lustre: Skipped 1 previous similar message [ 9553.407952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9553.425573] Lustre: Skipped 1 previous similar message [ 9558.501107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.519481] Lustre: Skipped 1 previous similar message [ 9559.169118] Lustre: server umount lustre-OST0000 complete [ 9563.617201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.630100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9563.631281] Lustre: Skipped 2 previous similar messages [ 9563.644693] Lustre: Skipped 1 previous similar message [ 9569.904972] Lustre: server umount lustre-MDT0000 complete [ 9573.818065] LustreError: 290000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447537 with bad export cookie 3143651517888726818 [ 9573.821250] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9573.831786] LustreError: 290000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9574.202762] Lustre: server umount lustre-MDT0001 complete [ 9589.238648] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 9594.369412] Key type lgssc unregistered [ 9594.989272] LNet: 293561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9595.007139] LNetError: 293561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9595.033493] LNet: Removed LNI 192.168.202.140@tcp [ 9596.071811] Key type .llcrypt unregistered [ 9596.075135] Key type ._llcrypt unregistered [ 9628.269040] Key type ._llcrypt registered [ 9628.275421] Key type .llcrypt registered [ 9628.480807] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 9650.299919] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 9652.127484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9652.337877] alg: No test for adler32 (adler32-zlib) [ 9653.733294] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9654.246036] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 9656.016802] Key type lgssc registered [ 9657.757986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9717.213950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9717.288447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.702125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9718.753133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9718.866742] Lustre: lustre-MDT0000: new disk, initializing [ 9719.085916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.129081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9725.316628] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9737.095390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9737.281394] Lustre: 297398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9737.359822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9737.367084] Lustre: Skipped 1 previous similar message [ 9737.454386] Lustre: lustre-MDT0001: new disk, initializing [ 9737.618352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9737.689424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9737.708629] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9743.433646] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9754.305542] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9761.214546] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.521182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.873618] Lustre: lustre-OST2710: new disk, initializing [ 9769.876973] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9769.887458] Lustre: 299366:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9770.065457] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9777.146787] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9778.814704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9778.831880] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9778.993183] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9787.391761] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9795.863859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9796.112225] Lustre: lustre-OST03e8: new disk, initializing [ 9796.117703] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9796.125964] Lustre: 300433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9796.341331] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9801.813174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9801.830207] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9801.889584] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9803.022937] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9811.753279] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9820.581674] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9820.848567] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9826.448561] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9826.743620] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9831.855832] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9832.236623] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9837.218399] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9837.449648] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9843.172554] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9843.184299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9843.190801] Lustre: Skipped 3 previous similar messages [ 9843.200415] Lustre: Skipped 2 previous similar messages [ 9844.151479] Lustre: server umount lustre-MDT0000 complete [ 9848.474355] LustreError: 297390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447812 with bad export cookie 11491255692667745492 [ 9848.477260] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9848.483712] LustreError: 297390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9848.776669] Lustre: server umount lustre-MDT0001 complete [ 9863.918977] Lustre: server umount lustre-OST2710 complete [ 9878.584513] Lustre: server umount lustre-OST03e8 complete [ 9905.836325] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 9920.477283] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 9972.208954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.467734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9972.484984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9972.531478] Lustre: lustre-MDT0000: new disk, initializing [ 9972.610924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.627115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9977.575665] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [ 9990.908428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.038176] Lustre: 305689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9991.115790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9991.120514] Lustre: Skipped 1 previous similar message [ 9991.207762] Lustre: lustre-MDT0001: new disk, initializing [ 9991.358056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9991.393727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9991.400878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9996.589295] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10008.949409] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10015.468996] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10023.552410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.786788] Lustre: lustre-OST0000: new disk, initializing [10023.789854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10023.800366] Lustre: 307659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10023.873414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10031.413955] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10031.721245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10031.730508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10031.961450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10043.032467] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10049.108717] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10049.382899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10055.330309] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10055.569630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10057.703479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10057.707197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.710658] LustreError: Skipped 1 previous similar message [10057.712974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10057.712981] Lustre: Skipped 1 previous similar message [10062.819457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10062.847194] Lustre: Skipped 2 previous similar messages [10063.736468] Lustre: server umount lustre-OST0000 complete [10073.060222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.070910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10073.084401] Lustre: Skipped 2 previous similar messages [10075.790366] Lustre: server umount lustre-MDT0000 complete [10078.178911] LustreError: 305702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10078.202856] LustreError: 305702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10079.362417] LustreError: 305681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448043 with bad export cookie 11491255692667747172 [10079.365875] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10079.379749] LustreError: 305681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10079.871889] Lustre: server umount lustre-MDT0001 complete [10089.919377] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:07:32 (1788448052) [10091.633409] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10094.076145] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:07:35 (1788448055) [10098.515233] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [10130.272025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.412796] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10130.452719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10130.557420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10135.013743] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10143.903248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.982019] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10144.013491] Lustre: 311518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10149.262658] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10172.722389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10172.963730] Lustre: Denying initial registration attempt from nid 192.168.202.140@tcp, specified as failover [10172.972122] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10172.978485] LustreError: 312976:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [10172.984371] LustreError: 312976:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10172.988412] LustreError: 312976:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10173.053493] Lustre: server umount lustre-OST0000 complete [10173.058429] LustreError: 312976:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10182.629240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10182.642335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10182.645991] Lustre: Skipped 1 previous similar message [10182.649526] Lustre: Skipped 1 previous similar message [10186.751722] Lustre: server umount lustre-MDT0000 complete [10187.745913] LustreError: 311526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.789152] LustreError: 311526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10191.689013] LustreError: 312262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448155 with bad export cookie 11491255692667748054 [10191.698710] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.701456] LustreError: 312262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10192.090291] Lustre: server umount lustre-MDT0001 complete [10210.999291] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [10216.931638] Key type lgssc unregistered [10217.488705] LNet: 314117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10217.503680] LNetError: 314117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10217.517736] LNet: Removed LNI 192.168.202.140@tcp [10218.660850] Key type .llcrypt unregistered [10218.662933] Key type ._llcrypt unregistered [10231.754670] Key type ._llcrypt registered [10231.758914] Key type .llcrypt registered [10231.943959] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [10233.705956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10233.742149] alg: No test for adler32 (adler32-zlib) [10234.800841] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10235.049255] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [10236.713493] Key type lgssc registered [10238.172408] Lustre: Echo OBD driver; http://www.lustre.org/ [10287.082276] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [10296.852220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10296.879916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10298.211715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10298.236894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10298.415822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10304.792797] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10313.602526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10313.806473] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10313.884276] Lustre: 316876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10314.212160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10318.447315] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10327.456877] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10333.696396] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10354.922717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10355.460052] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10355.751187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10364.492392] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10374.489412] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10382.314448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.316139] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10382.339695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10382.343046] LustreError: Skipped 1 previous similar message [10382.366979] Lustre: Skipped 1 previous similar message [10383.904935] Lustre: server umount lustre-OST0000 complete [10391.010142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10391.026731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.069677] Lustre: Skipped 1 previous similar message [10391.073934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.546960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.547812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10394.647381] Lustre: server umount lustre-MDT0000 complete [10399.429144] LustreError: 316867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448363 with bad export cookie 1853134740042298677 [10399.432991] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.458162] LustreError: 316867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10400.035788] Lustre: server umount lustre-MDT0001 complete [10413.845471] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [10417.124648] Key type lgssc unregistered [10417.449766] LNet: 320452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10417.462754] LNetError: 320452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10417.493817] LNet: Removed LNI 192.168.202.140@tcp [10418.632603] Key type .llcrypt unregistered [10418.635900] Key type ._llcrypt unregistered [10446.270853] Key type ._llcrypt registered [10446.272858] Key type .llcrypt registered [10446.395560] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [10447.847641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10447.887245] alg: No test for adler32 (adler32-zlib) [10448.969161] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10449.202841] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [10450.928596] Key type lgssc registered [10452.123307] Lustre: Echo OBD driver; http://www.lustre.org/ [10462.055669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10462.077134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.764289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10468.481711] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10473.970649] LustreError: 321871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10473.996949] LustreError: 321871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10478.295772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.172978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10484.103674] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10495.124961] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10502.686414] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10515.228775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.890142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10524.609055] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10533.142489] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10549.621529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [10549.731823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.749394] Lustre: Skipped 1 previous similar message [10553.092487] Lustre: server umount lustre-MDT0000 complete [10554.698918] LustreError: 321865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.203118] LustreError: 321851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448521 with bad export cookie 12469092762819656910 [10557.213561] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.219769] LustreError: 321851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10559.815266] LustreError: 321867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10559.827391] LustreError: 321867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10559.833784] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [10559.849176] Lustre: Skipped 3 previous similar messages [10559.975259] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10559.989161] Lustre: Skipped 1 previous similar message [10562.999615] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10562.999844] LustreError: 321865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.011657] Lustre: Skipped 1 previous similar message [10563.054560] LustreError: 321865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10563.799606] Lustre: server umount lustre-MDT0001 complete [10570.945783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10579.922395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10582.112202] LustreError: 325474:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10582.122774] LustreError: 325474:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000906af68a x1875324083157120/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788448546 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10582.150550] LustreError: 325474:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10582.513286] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xad0b1dcdbf429b6e [10582.531402] Lustre: MGC192.168.202.140@tcp: Connection restored to 0@lo (at 0@lo) [10582.895259] LustreError: 323703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10582.990800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10583.052526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10587.285864] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10592.240497] LustreError: 325490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10592.263835] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10592.277085] LustreError: 325490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10597.397628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10597.944607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10602.984626] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10604.553390] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10612.997476] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10614.849474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10622.568369] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10624.930423] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10633.707074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10633.709453] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.732598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10633.760089] Lustre: Skipped 1 previous similar message [10637.311856] Lustre: server umount lustre-OST0000 complete [10643.939168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.939888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10643.963605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10643.983082] Lustre: Skipped 2 previous similar messages [10647.710410] Lustre: server umount lustre-MDT0000 complete [10649.063163] LustreError: 325493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.089386] LustreError: 325493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10652.232206] LustreError: 321850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448616 with bad export cookie 12469092762819664750 [10652.237220] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10652.246484] LustreError: 321850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10652.607631] Lustre: server umount lustre-MDT0001 complete [10663.363796] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [10666.012696] Key type lgssc unregistered [10666.321550] LNet: 328163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10666.332424] LNetError: 328163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10666.357843] LNet: Removed LNI 192.168.202.140@tcp [10667.236984] Key type .llcrypt unregistered [10667.238921] Key type ._llcrypt unregistered [10700.037611] Key type ._llcrypt registered [10700.042591] Key type .llcrypt registered [10700.232174] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [10701.993498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10702.220926] alg: No test for adler32 (adler32-zlib) [10703.356181] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10703.619477] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [10705.305110] Key type lgssc registered [10706.638828] Lustre: Echo OBD driver; http://www.lustre.org/ [10716.366973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10716.387226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10718.038058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10723.140522] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10728.420779] LustreError: 329578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10728.445590] LustreError: 329578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10733.331982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10733.549035] LustreError: 329579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10733.978484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10738.756478] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10746.902527] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10753.929900] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10758.625559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10758.633445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10758.649240] Lustre: Skipped 1 previous similar message [10758.673866] Lustre: Skipped 1 previous similar message [10759.717237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.730416] Lustre: Skipped 1 previous similar message [10764.056596] Lustre: server umount lustre-MDT0000 complete [10764.775331] LustreError: 330582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10764.795115] LustreError: 330582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10768.238682] LustreError: 330313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448732 with bad export cookie 7631228569411078427 [10768.242975] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.248204] LustreError: 330313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10768.721759] Lustre: server umount lustre-MDT0001 complete [10784.862839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.134596] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10785.195240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10785.459478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10790.416827] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10799.859697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.967807] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10799.997929] Lustre: 332166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10800.343836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10805.206355] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10812.397875] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10818.895168] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10828.075767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10828.235065] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10828.243921] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10828.246957] LustreError: 334050:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10828.249420] LustreError: 334050:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10828.251440] LustreError: 334050:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10828.295791] Lustre: server umount lustre-OST0000 complete [10828.298456] LustreError: 334050:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10838.954441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10839.127490] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10839.380630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10844.848509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10846.977599] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10856.475247] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10864.636366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10864.730693] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10864.824364] Lustre: lustre-OST0001: new disk, initializing [10864.834163] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10864.847355] Lustre: 335594:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10865.000916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10870.439478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10870.454528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10870.568817] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10871.855773] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [10879.529750] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10886.122327] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10886.132692] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.140640] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10886.154707] Lustre: Skipped 1 previous similar message [10886.162332] Lustre: Skipped 1 previous similar message [10888.549325] Lustre: server umount lustre-OST0001 complete [10891.255042] LustreError: 334794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.276669] LustreError: 334794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10895.843216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10895.853562] LustreError: Skipped 1 previous similar message [10895.858136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10895.875092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10896.357017] LustreError: 334518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.377927] LustreError: 334518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10898.532676] Lustre: server umount lustre-OST0000 complete [10906.601575] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10906.609994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10906.626943] Lustre: Skipped 2 previous similar messages [10906.639192] Lustre: Skipped 2 previous similar messages [10908.986627] Lustre: server umount lustre-MDT0000 complete [10911.715425] LustreError: 332177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10911.734651] LustreError: 332177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10913.360968] LustreError: 332157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448877 with bad export cookie 7631228569411078980 [10913.376036] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10913.378093] LustreError: 332157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10914.159578] Lustre: server umount lustre-MDT0001 complete [10929.275512] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [10933.656808] Key type lgssc unregistered [10934.083225] LNet: 337551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10934.099796] LNetError: 337551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10934.124861] LNet: Removed LNI 192.168.202.140@tcp [10935.168760] Key type .llcrypt unregistered [10935.171272] Key type ._llcrypt unregistered [11007.978337] Key type ._llcrypt registered [11007.981254] Key type .llcrypt registered [11008.087087] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [11022.085886] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [11022.811430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11022.822590] alg: No test for adler32 (adler32-zlib) [11023.916533] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11024.128713] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [11025.800312] Key type lgssc registered [11026.803788] Lustre: Echo OBD driver; http://www.lustre.org/ [11062.073395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11062.131093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.409160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11063.451472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11063.556586] Lustre: lustre-MDT0000: new disk, initializing [11063.703344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.718494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11068.789750] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11079.795707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11079.918989] Lustre: 342479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11079.965247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11079.968548] Lustre: Skipped 1 previous similar message [11080.026916] Lustre: lustre-MDT0001: new disk, initializing [11080.108061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11080.142836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11080.156646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11085.180715] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11094.980858] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11100.899365] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11107.687908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.965451] Lustre: lustre-OST0000: new disk, initializing [11107.968391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11107.974394] Lustre: 344442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11108.053605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11114.225731] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11115.044425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11115.054322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11115.178976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11124.157496] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11130.748580] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11131.025049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11136.294392] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11136.493782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11141.089786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11141.111432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11141.114883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11141.143285] Lustre: Skipped 1 previous similar message [11146.224168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11146.228602] Lustre: Skipped 1 previous similar message [11151.333890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11151.344895] Lustre: Skipped 1 previous similar message [11152.864196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.301446] Lustre: server umount lustre-OST0000 complete [11161.573261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.575671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11161.591251] Lustre: Skipped 2 previous similar messages [11161.608812] Lustre: Skipped 1 previous similar message [11165.007456] Lustre: server umount lustre-MDT0000 complete [11170.379978] LustreError: 342469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449134 with bad export cookie 14607546529644653146 [11170.404763] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11170.407315] LustreError: 342469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11170.887666] Lustre: server umount lustre-MDT0001 complete [11181.891275] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:25:44 (1788449144) [11191.920443] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:25:53 (1788449153) [11201.974784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11202.513549] LustreError: 347204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.538023] LustreError: 347204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11202.603453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11207.657443] LustreError: 347205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11208.506979] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11212.777055] LustreError: 347204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11217.915644] LustreError: 347205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.091741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11218.519551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11223.754755] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11232.441904] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11238.607185] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11245.681097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11245.987708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11253.040733] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11261.024722] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11274.699684] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11276.850232] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11279.000291] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11281.279333] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11284.306679] Lustre: Failing over lustre-MDT0000 [11284.764205] Lustre: server umount lustre-MDT0000 complete [11284.963488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11284.980201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11285.020684] LustreError: 347201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.179266] LustreError: 348686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11294.195371] LustreError: 348686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11301.344448] Lustre: 340702:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449249/real 1788449249] req@00000000e2e68d75 x1875324686118016/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11301.365951] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11305.022146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.915116] LustreError: 347200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.934252] LustreError: 347200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [11311.601766] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcab86e2cb375b202 [11311.612596] Lustre: MGC192.168.202.140@tcp: Connection restored to 0@lo (at 0@lo) [11312.155202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11312.272726] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11316.045983] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11316.493790] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11317.228473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11317.245127] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11317.266980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11326.306503] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11328.245350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11332.552738] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11334.958324] Lustre: Failing over lustre-MDT0000 [11335.353652] Lustre: server umount lustre-MDT0000 complete [11337.697869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11337.715900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11337.739317] Lustre: Skipped 3 previous similar messages [11346.758627] LustreError: 347201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.774918] LustreError: 347201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11353.062190] Lustre: 340703:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449301/real 1788449301] req@00000000c0296741 x1875324686140288/t0(0) o400->MGC192.168.202.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11353.081717] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11356.099357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.312949] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcab86e2cb375b4fd [11363.334713] Lustre: MGC192.168.202.140@tcp: Connection restored to 0@lo (at 0@lo) [11363.344440] Lustre: Skipped 2 previous similar messages [11363.698661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11363.743040] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11367.250399] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11368.962241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11369.027509] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11369.108262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11369.571800] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11379.783828] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11381.387703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11389.436138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11389.440125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11389.456777] Lustre: Skipped 3 previous similar messages [11389.478073] Lustre: Skipped 2 previous similar messages [11395.462281] Lustre: server umount lustre-MDT0000 complete [11399.378161] LustreError: 351195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449363 with bad export cookie 14607546529644655869 [11399.380905] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11399.396083] LustreError: 351195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11399.663965] Lustre: server umount lustre-MDT0001 complete [11414.614169] Lustre: server umount lustre-OST0000 complete [11432.395917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11432.667661] LustreError: 353571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.679198] LustreError: 353571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [11432.727687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11437.524679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11447.684945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11454.935481] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11464.477244] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11470.508856] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11477.349298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11478.596722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11484.191568] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11493.095154] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11504.105690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11504.118979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11504.126809] Lustre: Skipped 3 previous similar messages [11509.225915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11510.403153] Lustre: server umount lustre-OST0000 complete [11521.718687] Lustre: server umount lustre-MDT0000 complete [11526.521063] LustreError: 353553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449490 with bad export cookie 14607546529644656611 [11526.528063] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.554923] LustreError: 353553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11527.167475] Lustre: server umount lustre-MDT0001 complete [11542.850978] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [11547.296688] Key type lgssc unregistered [11547.739030] LNet: 357103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11547.752724] LNetError: 357103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11548.774511] LNet: Removed LNI 192.168.202.140@tcp [11550.266489] Key type .llcrypt unregistered [11550.274123] Key type ._llcrypt unregistered [11581.404468] Key type ._llcrypt registered [11581.409351] Key type .llcrypt registered [11581.578710] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [11583.756183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11583.844156] alg: No test for adler32 (adler32-zlib) [11585.298776] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11585.613911] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [11587.370632] Key type lgssc registered [11589.346687] Lustre: Echo OBD driver; http://www.lustre.org/ [11600.075648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11600.091234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.766902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11608.222898] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11611.116458] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11611.133819] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11616.226780] LustreError: 358621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.846548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11619.426971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11625.033870] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11634.896857] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11642.060442] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11649.778259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11650.358712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11658.188661] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11667.295448] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11671.864482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11685.867410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11685.878283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11685.905641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.398505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.408554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.425414] Lustre: Skipped 1 previous similar message [11690.298845] Lustre: server umount lustre-MDT0000 complete [11692.519519] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11692.552505] LustreError: 358620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11694.682854] LustreError: 359355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449658 with bad export cookie 397027315529592872 [11694.697815] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11695.139784] Lustre: server umount lustre-MDT0001 complete [11710.150503] Lustre: server umount lustre-OST0000 complete [11726.649735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11727.304487] LustreError: 362353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.435281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11732.449974] LustreError: 362354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11733.524442] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11742.690374] LustreError: 362354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.725443] LustreError: 362354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11744.034737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.547151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11749.884279] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11759.632782] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11766.331190] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11774.176565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11774.505932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11782.161127] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11790.220984] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11797.053711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11816.422214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11816.428424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.459202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11816.466839] Lustre: Skipped 2 previous similar messages [11821.393845] Lustre: server umount lustre-MDT0000 complete [11822.565119] LustreError: 362348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.593757] LustreError: 362348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11825.626116] LustreError: 362335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449789 with bad export cookie 397027315529593978 [11825.645055] LustreError: 362335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11825.650740] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11826.196342] Lustre: server umount lustre-MDT0001 complete [11840.254215] Lustre: server umount lustre-OST0000 complete [11856.035873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.394282] LustreError: 366285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.453400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11861.091982] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11870.721197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.818477] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11883.271924] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11890.001315] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11898.616711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11899.073210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11899.082101] Lustre: Skipped 1 previous similar message [11901.079674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11907.799195] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [11917.066615] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11927.033852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11927.037681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11927.046959] LustreError: Skipped 1 previous similar message [11927.047829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11927.047835] Lustre: Skipped 3 previous similar messages [11927.093355] Lustre: Skipped 3 previous similar messages [11928.887430] Lustre: server umount lustre-OST0000 complete [11937.255417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11937.294073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11937.302766] Lustre: Skipped 2 previous similar messages [11940.723553] Lustre: server umount lustre-MDT0000 complete [11942.371083] LustreError: 366286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.403349] LustreError: 366286:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11945.167288] LustreError: 366267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449909 with bad export cookie 397027315529595000 [11945.174890] LustreError: 366267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11945.181976] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11946.003575] Lustre: server umount lustre-MDT0001 complete [11958.054242] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [11961.102845] Key type lgssc unregistered [11961.515585] LNet: 369816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11961.520695] LNetError: 369816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11962.534881] LNet: Removed LNI 192.168.202.140@tcp [11963.640352] Key type .llcrypt unregistered [11963.642477] Key type ._llcrypt unregistered [11994.964876] Key type ._llcrypt registered [11994.967820] Key type .llcrypt registered [11995.174718] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [11996.224681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11996.315058] alg: No test for adler32 (adler32-zlib) [11997.739278] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11998.055670] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [11999.752631] Key type lgssc registered [12001.506157] Lustre: Echo OBD driver; http://www.lustre.org/ [12009.363703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12009.400658] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12009.412348] LustreError: 371254:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12009.438305] LustreError: 371254:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12009.444930] LustreError: 371254:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12009.451398] LustreError: 371254:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12009.455930] LustreError: 371254:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12017.502028] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12017.513646] LustreError: 371653:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12017.537580] LustreError: 371653:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12017.542716] LustreError: 371653:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12017.548408] LustreError: 371653:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12017.554733] LustreError: 371653:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12039.337946] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [12042.362396] Key type lgssc unregistered [12042.661412] LNet: 372655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.667737] LNetError: 372655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.684954] LNet: Removed LNI 192.168.202.140@tcp [12043.555867] Key type .llcrypt unregistered [12043.565530] Key type ._llcrypt unregistered [12065.139510] Key type ._llcrypt registered [12065.142568] Key type .llcrypt registered [12065.242558] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [12080.082156] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [12081.549465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12081.642789] alg: No test for adler32 (adler32-zlib) [12082.795951] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12083.014627] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [12084.672565] Key type lgssc registered [12085.788446] Lustre: Echo OBD driver; http://www.lustre.org/ [12121.467700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12121.507702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12122.756332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12122.791787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12122.916460] Lustre: lustre-MDT0000: new disk, initializing [12123.054723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12123.151830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12127.763694] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12138.273384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12138.419680] Lustre: 375810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12138.518405] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12138.534352] Lustre: Skipped 1 previous similar message [12138.616423] Lustre: lustre-MDT0001: new disk, initializing [12138.720326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12138.777661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12138.803346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12143.038777] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12152.509500] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12157.715800] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12164.273687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12164.641907] Lustre: lustre-OST0000: new disk, initializing [12164.648500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12164.654145] Lustre: 377776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12164.749237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12171.593891] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12175.004022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12175.015889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12175.178119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12182.504857] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12188.330885] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12188.539519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12192.995098] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12193.183852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12195.815700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.822353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12195.836544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12195.841666] LustreError: Skipped 1 previous similar message [12195.853151] Lustre: Skipped 1 previous similar message [12200.930879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12200.945093] Lustre: Skipped 1 previous similar message [12201.644791] Lustre: server umount lustre-OST0000 complete [12206.055704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12206.066956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12206.075545] Lustre: Skipped 2 previous similar messages [12206.089849] Lustre: Skipped 1 previous similar message [12211.185840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12211.196306] Lustre: Skipped 1 previous similar message [12212.095237] Lustre: server umount lustre-MDT0000 complete [12216.597230] LustreError: 375801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450180 with bad export cookie 5232261984673905315 [12216.610423] LustreError: 375801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12216.615216] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12217.167544] Lustre: server umount lustre-MDT0001 complete [12226.776994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12228.992572] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:43:10 (1788450190) [12237.207728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12237.701425] LustreError: 380237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12237.726397] LustreError: 380237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12237.777961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12242.739858] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12242.916158] LustreError: 380238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12247.017879] LustreError: 380237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.459201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.757830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12256.472401] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12264.002340] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12269.143813] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12275.604349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12275.893969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12282.016607] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12288.573814] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12294.809378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12294.917329] Lustre: lustre-OST0001: new disk, initializing [12294.919903] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12294.923700] Lustre: 383072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12294.973966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12299.932827] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12300.881531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12300.888925] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12300.973685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12308.501504] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12315.487800] Lustre: lustre-OST0001: Not available for connect from 192.168.202.40@tcp (stopping) [12316.647017] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12316.652960] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12318.363470] Lustre: server umount lustre-OST0001 complete [12320.577784] LustreError: 382055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.886098] LustreError: 383505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.906263] LustreError: 383505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12337.126849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12337.133839] LustreError: 382055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.140464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12337.140472] Lustre: Skipped 2 previous similar messages [12337.148864] Lustre: Skipped 2 previous similar messages [12337.185496] LustreError: 382055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12348.384317] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12348.589093] Lustre: server umount lustre-OST0000 complete [12352.482388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.495099] Lustre: Skipped 1 previous similar message [12357.603665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.619128] Lustre: Skipped 8 previous similar messages [12358.793033] Lustre: server umount lustre-MDT0000 complete [12362.471106] LustreError: 380219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450326 with bad export cookie 5232261984673906197 [12362.476547] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12362.489421] LustreError: 380219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12368.851258] Lustre: server umount lustre-MDT0001 complete [12381.116511] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [12385.386628] Key type lgssc unregistered [12385.919079] LNet: 385062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12385.934847] LNetError: 385062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12385.954314] LNet: Removed LNI 192.168.202.140@tcp [12387.412264] Key type .llcrypt unregistered [12387.417577] Key type ._llcrypt unregistered [12435.523822] Key type ._llcrypt registered [12435.525708] Key type .llcrypt registered [12435.649283] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [12436.622373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12436.673069] alg: No test for adler32 (adler32-zlib) [12437.677386] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12437.889238] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [12439.536439] Key type lgssc registered [12440.546804] Lustre: Echo OBD driver; http://www.lustre.org/ [12447.647957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12447.672944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12448.916062] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12448.933971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12449.112202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12453.813236] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12463.020979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12463.193564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12463.242983] Lustre: 387856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12463.481714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12469.060304] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12476.471777] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12482.692146] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12485.602993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.627960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12485.633904] Lustre: Skipped 1 previous similar message [12490.721573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12490.729846] Lustre: Skipped 1 previous similar message [12491.383464] Lustre: server umount lustre-MDT0000 complete [12495.944840] LustreError: 387851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450460 with bad export cookie 17092320681912102615 [12495.963472] LustreError: 387851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12495.968552] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12496.327566] Lustre: server umount lustre-MDT0001 complete [12510.163347] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:47:52 (1788450472) [12518.475252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12519.110885] LustreError: 390371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.132229] LustreError: 390371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12519.226772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12524.520986] LustreError: 390371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12524.720304] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12529.634631] LustreError: 390372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.768660] LustreError: 390371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.925431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12535.595776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12541.997505] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12550.797668] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12556.912818] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12564.572407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12564.744795] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12565.072829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12570.551509] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12577.331412] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12585.075886] Lustre: Permanently deactivating lustre-OST0000 [12585.088687] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12591.178660] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12593.392473] LustreError: 393307:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12593.407673] LustreError: 393307:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12593.419348] LustreError: 393307:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12596.206065] LustreError: 393356:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12596.216996] LustreError: 393356:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12604.596715] Lustre: server umount lustre-OST0000 complete [12607.578646] LustreError: 393606:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12607.584614] LustreError: 393606:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12610.021795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12610.025947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12610.037465] Lustre: Skipped 2 previous similar messages [12610.050830] Lustre: Skipped 1 previous similar message [12615.137975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12615.143595] Lustre: Skipped 1 previous similar message [12620.259526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12620.275220] Lustre: Skipped 1 previous similar message [12624.352224] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12624.661535] Lustre: server umount lustre-MDT0000 complete [12625.378637] LustreError: 390372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.416140] LustreError: 390372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12628.777381] LustreError: 390352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450592 with bad export cookie 17092320681912103196 [12628.787856] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12628.794338] LustreError: 390352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12638.362934] Lustre: server umount lustre-MDT0001 complete [12649.160948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.723146] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12657.911078] LustreError: 394767:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12666.006413] Lustre: 395107:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.140@tcp [12668.544857] Lustre: server umount MGS complete [12681.574271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12682.256095] LustreError: 395831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12682.288947] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12682.297735] Lustre: Skipped 1 previous similar message [12682.381279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12688.176884] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12697.907794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12698.113374] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12703.455677] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12712.410288] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12719.935810] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12722.382359] Lustre: Permanently reactivating lustre-OST0000 [12722.404715] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12722.409435] Lustre: Skipped 2 previous similar messages [12748.770653] LustreError: 397564:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12759.361569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12759.702663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12759.718040] Lustre: Skipped 1 previous similar message [12767.359157] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12775.791954] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12783.832928] Lustre: server umount lustre-OST0000 complete [12789.730039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.735652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12789.751818] Lustre: Skipped 1 previous similar message [12789.765338] Lustre: Skipped 1 previous similar message [12799.067870] Lustre: server umount lustre-MDT0000 complete [12799.969158] Lustre: 387181:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450748/real 1788450748] req@000000008fa4f627 x1875326167777664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788450764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12799.969263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12799.969472] LustreError: 397018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.969480] LustreError: 397018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12799.992880] Lustre: 387181:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12803.715094] LustreError: 396565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450767 with bad export cookie 17092320681912104246 [12803.717635] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12803.730747] LustreError: 396565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12804.277218] Lustre: server umount lustre-MDT0001 complete [12816.520505] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [12819.629886] Key type lgssc unregistered [12819.953791] LNet: 400150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12819.976245] LNetError: 400150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12819.997408] LNet: Removed LNI 192.168.202.140@tcp [12820.905252] Key type .llcrypt unregistered [12820.907759] Key type ._llcrypt unregistered [12840.135294] Key type ._llcrypt registered [12840.142077] Key type .llcrypt registered [12840.292561] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [12861.579444] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [12862.770882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12862.926766] alg: No test for adler32 (adler32-zlib) [12864.086101] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12864.312753] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [12865.978216] Key type lgssc registered [12867.135417] Lustre: Echo OBD driver; http://www.lustre.org/ [12901.865509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12901.903093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12903.114466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12903.141167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12903.207152] Lustre: lustre-MDT0000: new disk, initializing [12903.298348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12903.327516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12907.809352] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12918.541970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12918.735511] Lustre: 403303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12918.859231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12918.863759] Lustre: Skipped 1 previous similar message [12919.003178] Lustre: lustre-MDT0001: new disk, initializing [12919.180481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12919.267377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12919.277089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12923.854555] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12934.576977] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12941.748523] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12948.999038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.321846] Lustre: lustre-OST0000: new disk, initializing [12949.325371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12949.332606] Lustre: 405270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12949.429938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12957.295886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12957.302491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12957.407435] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [12957.517481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12968.308907] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12972.828685] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12973.000160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12977.594250] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12977.799292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12983.267954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12983.278222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12983.293603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12983.299072] Lustre: Skipped 1 previous similar message [12983.319616] Lustre: Skipped 1 previous similar message [12985.830659] Lustre: server umount lustre-OST0000 complete [12990.955712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12990.971676] LustreError: Skipped 1 previous similar message [12990.973773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12990.995966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12993.507216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12993.513554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12998.626940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12998.636653] Lustre: Skipped 2 previous similar messages [13003.749144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13003.756976] Lustre: Skipped 1 previous similar message [13004.256197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13004.539360] Lustre: server umount lustre-MDT0000 complete [13009.248058] LustreError: 403294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450973 with bad export cookie 17426447210989970878 [13009.265324] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13009.764757] Lustre: server umount lustre-MDT0001 complete [13019.895610] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13021.685481] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:56:24 (1788450984) [13030.284676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13030.681969] LustreError: 407737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13030.713882] LustreError: 407737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13030.817923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13036.008442] LustreError: 407736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13037.232504] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13040.101688] LustreError: 407737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13045.218463] LustreError: 407736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.732920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13050.527944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.920623] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13066.992978] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13074.003188] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13082.792058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13083.293169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13092.546055] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13103.305378] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13105.524041] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13108.735882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13116.396210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13116.399494] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13116.424157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13121.066873] Lustre: server umount lustre-OST0000 complete [13126.625444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13126.633213] Lustre: Skipped 1 previous similar message [13132.066653] Lustre: server umount lustre-MDT0000 complete [13136.477348] LustreError: 407716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451100 with bad export cookie 17426447210989971760 [13136.503099] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13136.822501] Lustre: server umount lustre-MDT0001 complete [13148.093678] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13151.160910] Key type lgssc unregistered [13151.421897] LNet: 411466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13151.434021] LNetError: 411466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13152.489624] LNet: Removed LNI 192.168.202.140@tcp [13153.279787] Key type .llcrypt unregistered [13153.281375] Key type ._llcrypt unregistered [13180.891982] Key type ._llcrypt registered [13180.896481] Key type .llcrypt registered [13181.023651] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13181.444322] Key type .llcrypt unregistered [13181.447887] Key type ._llcrypt unregistered [13201.298694] Key type ._llcrypt registered [13201.300848] Key type .llcrypt registered [13201.438941] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13202.862083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13202.988486] alg: No test for adler32 (adler32-zlib) [13204.088318] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13204.383556] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [13206.088764] Key type lgssc registered [13206.910317] Lustre: Echo OBD driver; http://www.lustre.org/ [13213.217956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13213.240945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13214.672701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13219.204433] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13224.936502] LustreError: 413673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13224.970466] LustreError: 413673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13230.068098] LustreError: 413672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13231.163106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13231.450336] LustreError: 413673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13231.474218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13236.710407] LustreError: 413672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.751332] LustreError: 413672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13238.049904] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13239.787363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13244.922383] LustreError: 413672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13244.940161] LustreError: 413672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13247.776508] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13254.927828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13255.143637] LustreError: 413673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13255.178053] LustreError: 413673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13255.444324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13260.174765] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13270.498387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13270.512749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13270.532119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13271.024273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13271.034673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13271.054757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13271.060882] Lustre: Skipped 1 previous similar message [13271.752752] Lustre: server umount lustre-OST0000 complete [13276.129904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13276.141738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13276.148711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13276.185910] Lustre: Skipped 1 previous similar message [13277.701143] Lustre: server umount lustre-MDT0000 complete [13281.192370] LustreError: 413653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451245 with bad export cookie 15955434144768133379 [13281.203417] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13281.209015] LustreError: 413653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13281.619627] Lustre: server umount lustre-MDT0001 complete [13293.915791] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13296.938600] Key type lgssc unregistered [13297.255875] LNet: 416817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13297.273908] LNetError: 416817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13297.331261] LNet: Removed LNI 192.168.202.140@tcp [13298.627633] Key type .llcrypt unregistered [13298.629734] Key type ._llcrypt unregistered [13326.030778] Key type ._llcrypt registered [13326.032522] Key type .llcrypt registered [13326.193952] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13327.646371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13328.097134] alg: No test for adler32 (adler32-zlib) [13329.191607] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13329.626679] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [13331.360226] Key type lgssc registered [13332.830518] Lustre: Echo OBD driver; http://www.lustre.org/ [13341.534753] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13341.567687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13357.216761] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13362.401929] LustreError: 418218:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.140@tcp: failed processing log, type 4: rc = -110 [13389.088749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13395.951505] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13404.910603] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13412.912802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13413.581131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13413.683709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13418.985852] LustreError: 418244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.005018] LustreError: 418244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13419.483404] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13423.085711] LustreError: 418243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13423.105563] LustreError: 418243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13428.211485] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.247847] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13430.430591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13430.867971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13436.075329] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13444.635255] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13451.954656] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13461.485596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13461.494537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13461.501362] Lustre: Skipped 1 previous similar message [13461.514074] Lustre: Skipped 1 previous similar message [13466.593500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13466.616233] Lustre: Skipped 1 previous similar message [13467.653520] Lustre: server umount lustre-OST0000 complete [13474.505253] Lustre: server umount lustre-MDT0000 complete [13476.833653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13476.835172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13476.856532] LustreError: 419261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13476.896595] Lustre: Skipped 1 previous similar message [13479.405552] LustreError: 418226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451443 with bad export cookie 11811052286107439622 [13479.407523] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13479.415811] LustreError: 418226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13479.730243] Lustre: server umount lustre-MDT0001 complete [13492.297956] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13496.326507] Key type lgssc unregistered [13496.606928] LNet: 421791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13496.621286] LNetError: 421791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13496.642364] LNet: Removed LNI 192.168.202.140@tcp [13497.916296] Key type .llcrypt unregistered [13497.918719] Key type ._llcrypt unregistered [13528.824493] Key type ._llcrypt registered [13528.826778] Key type .llcrypt registered [13528.976585] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13530.664039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13530.930541] alg: No test for adler32 (adler32-zlib) [13532.087179] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13532.459868] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [13534.208244] Key type lgssc registered [13536.079393] Lustre: Echo OBD driver; http://www.lustre.org/ [13544.021722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13544.041270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13545.489727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13549.631788] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13555.695246] LustreError: 423209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13555.715568] LustreError: 423209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13560.797240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13560.802972] LustreError: 423210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13561.184410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13567.232147] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13577.193949] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13584.216519] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13591.699770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13592.182200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13598.970190] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13608.339876] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13615.732889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13622.756519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13622.773033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13622.792960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13624.686364] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [13625.149019] Lustre: server umount lustre-MDT0000 complete [13626.354197] LustreError: 423210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13626.365198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13627.720656] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.40@tcp arrived at 1788451591 with bad export cookie 6574858205719850085 [13627.739329] LustreError: 423189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13629.768154] LustreError: 423190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451593 with bad export cookie 6574858205719849420 [13629.778452] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13629.781449] LustreError: 423190:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13631.457919] LustreError: 424149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13631.483431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13631.491682] LustreError: 424149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13631.518387] Lustre: Skipped 1 previous similar message [13631.538562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13634.466522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13639.589707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13639.591131] LustreError: 423205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.639863] LustreError: 423205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13644.256155] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13644.502359] Lustre: server umount lustre-OST0000 complete [13649.901942] LustreError: 424149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13649.921986] LustreError: 424149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13653.653445] Lustre: server umount lustre-MDT0001 complete [13667.409914] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13670.614648] Key type lgssc unregistered [13670.985688] LNet: 426916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13670.999404] LNetError: 426916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13671.011810] LNet: Removed LNI 192.168.202.140@tcp [13671.734845] Key type .llcrypt unregistered [13671.737790] Key type ._llcrypt unregistered [13704.024747] Key type ._llcrypt registered [13704.031692] Key type .llcrypt registered [13704.131977] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13705.092970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13705.225378] alg: No test for adler32 (adler32-zlib) [13706.618416] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13707.054974] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [13708.824653] Key type lgssc registered [13710.468936] Lustre: Echo OBD driver; http://www.lustre.org/ [13720.330876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13720.359278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13721.995299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13726.666568] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13732.334461] LustreError: 428329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13732.356962] LustreError: 428329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13733.358295] LustreError: 428330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.049450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13737.529379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13743.091514] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13752.622158] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13758.987831] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13766.525169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13766.931491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13768.962686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13773.446467] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13781.625549] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13787.637666] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13787.649474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13787.652136] Lustre: Skipped 1 previous similar message [13789.557123] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [13792.455501] Lustre: server umount lustre-MDT0001 complete [13792.740691] LustreError: 428324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.863233] LustreError: 430818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13797.886020] LustreError: 430818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13799.915330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13799.923575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13799.941123] Lustre: Skipped 1 previous similar message [13799.945674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13802.981714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13802.984865] LustreError: 428330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13805.414500] Lustre: server umount lustre-OST0000 complete [13810.420428] Lustre: server umount lustre-MDT0000 complete [13827.890095] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [13831.564136] Key type lgssc unregistered [13832.018184] LNet: 432014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13832.023957] LNetError: 432014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13832.049168] LNet: Removed LNI 192.168.202.140@tcp [13833.501187] Key type .llcrypt unregistered [13833.503844] Key type ._llcrypt unregistered [13862.729752] Key type ._llcrypt registered [13862.732961] Key type .llcrypt registered [13862.900715] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [13879.840452] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13881.099993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13881.159606] alg: No test for adler32 (adler32-zlib) [13882.560883] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13882.800098] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [13884.488170] Key type lgssc registered [13885.693211] Lustre: Echo OBD driver; http://www.lustre.org/ [13932.173263] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [13950.099937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13950.131862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.513939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13951.542590] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13951.700231] Lustre: lustre-MDT0000: new disk, initializing [13951.790942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13951.839567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13957.120433] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13974.627441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13974.989904] Lustre: lustre-OST0000: new disk, initializing [13974.999986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13975.008756] Lustre: Skipped 1 previous similar message [13975.018729] Lustre: 437052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13975.113284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13982.762821] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [13983.336137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13983.347964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13983.412126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13993.117044] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14011.477894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14011.635895] Lustre: 436034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14011.715429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14011.786539] Lustre: lustre-MDT0001: new disk, initializing [14011.909278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14011.967923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14011.984369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14017.423953] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14027.450814] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14027.648302] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14032.838454] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14033.046390] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14038.512591] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14038.515798] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14038.568724] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14038.580736] Lustre: Skipped 1 previous similar message [14042.966558] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [14044.630434] Lustre: server umount lustre-MDT0001 complete [14053.864538] LustreError: 436046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14053.877292] LustreError: 436046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14058.984862] LustreError: 436042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14059.028889] LustreError: 436042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14064.103657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14064.111946] LustreError: 436046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.128544] Lustre: Skipped 1 previous similar message [14064.131222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14064.131234] Lustre: Skipped 2 previous similar messages [14068.969938] Lustre: server umount lustre-OST0000 complete [14069.227258] LustreError: 439627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14078.920486] Lustre: server umount lustre-MDT0000 complete [14095.266666] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [14098.211881] Key type lgssc unregistered [14098.509679] LNet: 440803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14098.519750] LNetError: 440803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14098.533949] LNet: Removed LNI 192.168.202.140@tcp [14099.448862] Key type .llcrypt unregistered [14099.463783] Key type ._llcrypt unregistered [14142.677929] Key type ._llcrypt registered [14142.679936] Key type .llcrypt registered [14142.797621] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [14143.018306] Key type .llcrypt unregistered [14143.020125] Key type ._llcrypt unregistered [14161.068258] Key type ._llcrypt registered [14161.070292] Key type .llcrypt registered [14161.173795] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [14162.421627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14162.533089] alg: No test for adler32 (adler32-zlib) [14163.735187] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14163.985848] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [14165.672951] Key type lgssc registered [14166.984962] Lustre: Echo OBD driver; http://www.lustre.org/ [14195.187766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14195.207864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14196.507259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14196.537405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14196.633531] Lustre: lustre-MDT0000: new disk, initializing [14196.696751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14196.721222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14202.348786] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14216.177234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14216.314573] Lustre: 444175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14216.362981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14216.366493] Lustre: Skipped 1 previous similar message [14216.420380] Lustre: lustre-MDT0001: new disk, initializing [14216.504782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14216.550089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14216.561232] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14221.681995] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14230.759974] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14237.183574] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14244.710428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14245.139973] Lustre: lustre-OST0000: new disk, initializing [14245.143496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14245.149933] Lustre: 446147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14245.270583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14251.081767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14251.091593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14251.242215] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14252.343701] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14262.186566] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14271.987656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14271.991623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14272.012897] Lustre: Skipped 2 previous similar messages [14272.044446] Lustre: Skipped 2 previous similar messages [14274.798297] Lustre: server umount lustre-MDT0000 complete [14279.742646] LustreError: 446145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452243 with bad export cookie 10972090809574535521 [14279.749166] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14279.756850] LustreError: 446145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14280.093751] Lustre: server umount lustre-MDT0001 complete [14294.743096] Lustre: server umount lustre-OST0000 complete [14304.840743] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:17:47 (1788452267) [14312.522841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14312.970656] LustreError: 448173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14312.979393] LustreError: 448173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14313.030513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14317.117297] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14318.049201] LustreError: 448174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14318.083662] LustreError: 448174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14323.169923] LustreError: 448173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14325.487461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14326.040024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14331.918817] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14340.669338] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14347.357255] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14368.685559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14368.981424] Lustre: Found index 0 for lustre-OST0000, updating log [14369.307677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14376.159420] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [14383.024804] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14384.707750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14394.857046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14394.866710] Lustre: Skipped 1 previous similar message [14394.871517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14394.878642] Lustre: Skipped 1 previous similar message [14395.944889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14395.964183] Lustre: Skipped 1 previous similar message [14398.397991] Lustre: server umount lustre-OST0000 complete [14417.378063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14417.382492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14417.390857] Lustre: Skipped 1 previous similar message [14417.403060] Lustre: Skipped 1 previous similar message [14419.910670] Lustre: server umount lustre-MDT0000 complete [14422.498553] LustreError: 448173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14422.511334] LustreError: 448173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14423.273643] LustreError: 448155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452387 with bad export cookie 10972090809574536914 [14423.277568] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14423.283574] LustreError: 448155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14423.701693] Lustre: server umount lustre-MDT0001 complete [14426.145726] LNet: 1 local NIs in recovery (showing 1): 192.168.202.140@tcp [14432.622607] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:19:55 (1788452395) [15057.768852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15058.426259] LustreError: 454589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15058.566654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15063.273521] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15064.039928] LustreError: 454590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15068.203990] LustreError: 454589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15072.012423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15072.332100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15078.494767] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15089.266176] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15101.257129] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15118.368569] LNet: 1 local NIs in recovery (showing 1): 192.168.202.140@tcp [15124.459952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15124.677739] Lustre: Found index 0 for lustre-OST0000, updating log [15124.907550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15132.187543] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15141.138960] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15143.871597] Lustre: server umount lustre-OST0000 complete [15149.048962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15149.057961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15149.076793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.115133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15150.117877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.159336] Lustre: Skipped 1 previous similar message [15151.221068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15151.242666] Lustre: Skipped 1 previous similar message [15153.321560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15153.326110] Lustre: Skipped 3 previous similar messages [15154.679699] Lustre: server umount lustre-MDT0000 complete [15155.363995] LustreError: 455317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15155.383340] LustreError: 455317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15159.278972] LustreError: 454571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453123 with bad export cookie 10972090809574537999 [15159.289481] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15159.292271] LustreError: 454571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15159.592207] Lustre: server umount lustre-MDT0001 complete [15178.786110] LNet: 1 local NIs in recovery (showing 1): 192.168.202.140@tcp [15188.441860] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [15196.917934] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [15234.722557] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:33:17 (1788453197) [15251.526957] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [15254.427868] Key type lgssc unregistered [15254.768566] LNet: 461200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15254.777423] LNetError: 461200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15385.959393] LNet: Removed LNI 192.168.202.140@tcp [15386.806286] Key type .llcrypt unregistered [15386.811091] Key type ._llcrypt unregistered [15405.659660] Key type ._llcrypt registered [15405.661519] Key type .llcrypt registered [15405.800369] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [15407.240138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15407.328158] alg: No test for adler32 (adler32-zlib) [15408.456302] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15408.644699] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [15410.304801] Key type lgssc registered [15411.490310] Lustre: Echo OBD driver; http://www.lustre.org/ [15417.021463] LNet: Added LNI 192.168.202.140@tcp100 [8/256/0/180] [15417.942894] LNet: Added LNI 192.168.202.140@tcp101 [8/256/0/180] [15418.790705] LNet: Added LNI 192.168.202.140@tcp102 [8/256/0/180] [15419.566393] LNet: Added LNI 192.168.202.140@tcp103 [8/256/0/180] [15420.545403] LNet: Added LNI 192.168.202.140@tcp104 [8/256/0/180] [15421.468745] LNet: Added LNI 192.168.202.140@tcp105 [8/256/0/180] [15422.242693] LNet: Added LNI 192.168.202.140@tcp106 [8/256/0/180] [15423.118959] LNet: Added LNI 192.168.202.140@tcp107 [8/256/0/180] [15423.983838] LNet: Added LNI 192.168.202.140@tcp108 [8/256/0/180] [15424.973335] LNet: Added LNI 192.168.202.140@tcp109 [8/256/0/180] [15425.841408] LNet: Added LNI 192.168.202.140@tcp110 [8/256/0/180] [15426.763343] LNet: Added LNI 192.168.202.140@tcp111 [8/256/0/180] [15427.640720] LNet: Added LNI 192.168.202.140@tcp112 [8/256/0/180] [15428.466914] LNet: Added LNI 192.168.202.140@tcp113 [8/256/0/180] [15429.400775] LNet: Added LNI 192.168.202.140@tcp114 [8/256/0/180] [15430.260109] LNet: Added LNI 192.168.202.140@tcp115 [8/256/0/180] [15431.103094] LNet: Added LNI 192.168.202.140@tcp116 [8/256/0/180] [15431.922396] LNet: Added LNI 192.168.202.140@tcp117 [8/256/0/180] [15432.808701] LNet: Added LNI 192.168.202.140@tcp118 [8/256/0/180] [15433.551638] LNet: Added LNI 192.168.202.140@tcp119 [8/256/0/180] [15434.492054] LNet: Added LNI 192.168.202.140@tcp120 [8/256/0/180] [15435.365727] LNet: Added LNI 192.168.202.140@tcp121 [8/256/0/180] [15436.382831] LNet: Added LNI 192.168.202.140@tcp122 [8/256/0/180] [15437.276171] LNet: Added LNI 192.168.202.140@tcp123 [8/256/0/180] [15438.164666] LNet: Added LNI 192.168.202.140@tcp124 [8/256/0/180] [15439.208839] LNet: Added LNI 192.168.202.140@tcp125 [8/256/0/180] [15440.238976] LNet: Added LNI 192.168.202.140@tcp126 [8/256/0/180] [15441.116750] LNet: Added LNI 192.168.202.140@tcp127 [8/256/0/180] [15442.188606] LNet: Added LNI 192.168.202.140@tcp128 [8/256/0/180] [15443.044565] LNet: Added LNI 192.168.202.140@tcp129 [8/256/0/180] [15443.882835] LNet: Added LNI 192.168.202.140@tcp130 [8/256/0/180] [15444.790218] LNet: Added LNI 192.168.202.140@tcp131 [8/256/0/180] [15445.910836] LNet: Added LNI 192.168.202.140@tcp132 [8/256/0/180] [15446.855406] LNet: Added LNI 192.168.202.140@tcp133 [8/256/0/180] [15447.792450] LNet: Added LNI 192.168.202.140@tcp134 [8/256/0/180] [15448.587897] LNet: Added LNI 192.168.202.140@tcp135 [8/256/0/180] [15455.694355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15455.714155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15456.889914] Lustre: 464232:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.140@tcp: can't add NID 192.168.202.140@tcp130: rc = -75 [15456.920462] LNetError: 461691:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15456.959593] LNetError: Protocol error connecting to 192.168.202.140@tcp112 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15456.971411] LNetError: Skipped 3 previous similar messages [15457.084615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15457.103473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15457.156360] Lustre: lustre-MDT0000: new disk, initializing [15457.219266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15457.233790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15457.834890] LNet: 3 local NIs in recovery (showing 3): 192.168.202.140@tcp126, 192.168.202.140@tcp128, 192.168.202.140@tcp129 [15457.837074] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15457.867722] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15457.880819] LNetError: Protocol error connecting to 192.168.202.140@tcp100 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15457.885227] LNetError: Skipped 3 previous similar messages [15461.522282] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15474.076780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15474.415271] Lustre: 464250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15474.549572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15474.555312] Lustre: Skipped 1 previous similar message [15474.710467] Lustre: lustre-MDT0001: new disk, initializing [15474.768613] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.140@tcp: can't add NID 192.168.202.140@tcp130: rc = -75 [15474.784789] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15475.037354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15475.119181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15475.132880] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15481.516782] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15487.526609] LNetError: 464256:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15487.532733] LNetError: Protocol error connecting to 192.168.202.140@tcp125 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15490.595572] LNetError: 464260:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15490.616740] LNetError: 464260:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15490.622842] LNetError: Protocol error connecting to 192.168.202.140@tcp123 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15490.634648] LNetError: Skipped 1 previous similar message [15492.488213] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15495.716499] LNetError: 464254:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15495.735633] LNetError: 464254:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15495.761356] LNetError: Protocol error connecting to 192.168.202.140@tcp124 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15495.791903] LNetError: Skipped 2 previous similar messages [15499.204715] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15503.915782] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15503.925476] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15503.931136] LNetError: Protocol error connecting to 192.168.202.140@tcp123 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15503.937474] LNetError: Skipped 3 previous similar messages [15506.358330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15506.632212] Lustre: lustre-OST0000: new disk, initializing [15506.635743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15506.644652] Lustre: 466225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15506.728410] Lustre: 466230:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.140@tcp: can't add NID 192.168.202.140@tcp130: rc = -75 [15506.737693] Lustre: 466230:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15506.788616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15513.645283] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15516.252983] Lustre: 466684:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.140@tcp: can't add NID 192.168.202.140@tcp130: rc = -75 [15516.269491] Lustre: 466684:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15516.391592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15516.407995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15516.567519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15521.316163] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15521.325739] LNetError: 461692:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15521.334829] LNetError: Protocol error connecting to 192.168.202.140@tcp124 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15521.342932] LNetError: Skipped 9 previous similar messages [15522.205439] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15532.016973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15532.023347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15532.025979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15532.037486] Lustre: Skipped 1 previous similar message [15532.062479] Lustre: Skipped 1 previous similar message [15537.126422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15537.136457] Lustre: Skipped 1 previous similar message [15542.242131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15542.251303] Lustre: Skipped 1 previous similar message [15544.288144] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15544.494531] Lustre: server umount lustre-OST0000 complete [15551.969144] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15551.977132] LustreError: Skipped 1 previous similar message [15551.981578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15551.992960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15554.081813] LNetError: 461690:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.140 [15554.087400] LNetError: 461690:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15554.093747] LNetError: Protocol error connecting to 192.168.202.140@tcp123 at host 192.168.202.140:988: is it running a compatible version of Lustre? [15554.108781] LNetError: Skipped 10 previous similar messages [15554.883609] Lustre: server umount lustre-MDT0000 complete [15558.644337] LustreError: 464241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453522 with bad export cookie 1130489686027542174 [15558.659544] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15559.003801] Lustre: server umount lustre-MDT0001 complete [15568.640633] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [15571.417715] Key type lgssc unregistered [15571.895452] LNet: 468138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15571.903109] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15571.909182] LNet: Removed LNI 192.168.202.140@tcp [15572.966027] LNet: Removed LNI 192.168.202.140@tcp100 [15572.974428] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15572.983302] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15574.048234] LNet: Removed LNI 192.168.202.140@tcp101 [15574.053911] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15574.060909] LNet: Removed LNI 192.168.202.140@tcp102 [15575.136190] LNet: Removed LNI 192.168.202.140@tcp103 [15576.162546] LNet: Removed LNI 192.168.202.140@tcp104 [15576.192498] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15576.204461] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15576.216989] LNet: Removed LNI 192.168.202.140@tcp105 [15577.248498] LNet: Removed LNI 192.168.202.140@tcp106 [15578.272474] LNet: Removed LNI 192.168.202.140@tcp107 [15579.296202] LNet: Removed LNI 192.168.202.140@tcp108 [15580.320348] LNet: Removed LNI 192.168.202.140@tcp109 [15580.324534] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15580.335751] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15580.351562] LNet: Removed LNI 192.168.202.140@tcp110 [15581.408175] LNet: Removed LNI 192.168.202.140@tcp111 [15582.432196] LNet: Removed LNI 192.168.202.140@tcp112 [15583.456220] LNet: Removed LNI 192.168.202.140@tcp113 [15584.480356] LNet: Removed LNI 192.168.202.140@tcp114 [15585.504129] LNet: Removed LNI 192.168.202.140@tcp115 [15586.528176] LNet: Removed LNI 192.168.202.140@tcp116 [15587.552141] LNet: Removed LNI 192.168.202.140@tcp117 [15588.576295] LNet: Removed LNI 192.168.202.140@tcp118 [15588.594710] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15588.609618] LNetError: 468138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15588.618787] LNet: Removed LNI 192.168.202.140@tcp119 [15589.664253] LNet: Removed LNI 192.168.202.140@tcp120 [15590.688155] LNet: Removed LNI 192.168.202.140@tcp121 [15591.712348] LNet: Removed LNI 192.168.202.140@tcp122 [15592.736342] LNet: Removed LNI 192.168.202.140@tcp123 [15593.761062] LNet: Removed LNI 192.168.202.140@tcp124 [15594.784988] LNet: Removed LNI 192.168.202.140@tcp125 [15595.810457] LNet: Removed LNI 192.168.202.140@tcp126 [15596.833315] LNet: Removed LNI 192.168.202.140@tcp127 [15597.856769] LNet: Removed LNI 192.168.202.140@tcp128 [15598.880379] LNet: Removed LNI 192.168.202.140@tcp129 [15598.887377] LNet: Removed LNI 192.168.202.140@tcp130 [15598.893852] LNet: Removed LNI 192.168.202.140@tcp131 [15598.900084] LNet: Removed LNI 192.168.202.140@tcp132 [15598.907095] LNet: Removed LNI 192.168.202.140@tcp133 [15598.914381] LNet: Removed LNI 192.168.202.140@tcp134 [15598.930683] LNet: Removed LNI 192.168.202.140@tcp135 [15599.772311] Key type .llcrypt unregistered [15599.774939] Key type ._llcrypt unregistered [15629.676093] Key type ._llcrypt registered [15629.679869] Key type .llcrypt registered [15629.883489] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [15650.115855] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [15653.066218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15653.093750] alg: No test for adler32 (adler32-zlib) [15654.286068] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15654.449451] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [15656.080606] Key type lgssc registered [15658.062881] Lustre: Echo OBD driver; http://www.lustre.org/ [15700.712679] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:41:02 (1788453662) [15722.081340] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [15724.788731] Key type lgssc unregistered [15725.131719] LNet: 472227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15725.143146] LNetError: 472227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15725.169811] LNet: Removed LNI 192.168.202.140@tcp [15725.998755] Key type .llcrypt unregistered [15726.003086] Key type ._llcrypt unregistered [15742.346348] Key type ._llcrypt registered [15742.348283] Key type .llcrypt registered [15742.474774] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [15744.103609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15744.204248] alg: No test for adler32 (adler32-zlib) [15745.300560] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15745.583742] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [15747.240191] Key type lgssc registered [15748.802236] Lustre: Echo OBD driver; http://www.lustre.org/ [15759.163247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15759.198663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15760.543547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15760.564972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15760.618184] Lustre: lustre-MDT0000: new disk, initializing [15760.677963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15760.690581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15765.961158] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15778.356030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15778.458769] Lustre: 473439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15778.503423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15778.506879] Lustre: Skipped 1 previous similar message [15778.558525] Lustre: lustre-MDT0001: new disk, initializing [15778.645464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15778.697848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15778.707714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15784.343868] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15793.613984] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15802.381567] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15812.263348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15812.730428] Lustre: lustre-OST0000: new disk, initializing [15812.740430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15812.760310] Lustre: 475405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15812.927340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15820.928454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15820.942710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15821.266488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15821.782596] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [15833.113436] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15837.964703] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing lnet_if_list [15844.623420] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15844.680935] LNet: Added LNI 192.168.202.140@tcp10 [8/256/0/180] [15850.899689] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15850.926957] LNet: Added LNI 192.168.202.140@tcp11 [8/256/0/180] [15855.737662] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15855.763807] LNet: Added LNI 192.168.202.140@tcp12 [8/256/0/180] [15860.330385] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15860.361849] LNet: Added LNI 192.168.202.140@tcp13 [8/256/0/180] [15864.865393] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15864.915537] LNet: Added LNI 192.168.202.140@tcp14 [8/256/0/180] [15870.136914] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15870.159918] LNet: Added LNI 192.168.202.140@tcp15 [8/256/0/180] [15875.757877] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15875.774102] LNet: Added LNI 192.168.202.140@tcp16 [8/256/0/180] [15881.860196] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15881.879465] LNet: Added LNI 192.168.202.140@tcp17 [8/256/0/180] [15887.510889] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15887.540739] LNet: Added LNI 192.168.202.140@tcp18 [8/256/0/180] [15893.473692] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15893.518557] LNet: Added LNI 192.168.202.140@tcp19 [8/256/0/180] [15899.133596] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15899.165648] LNet: Added LNI 192.168.202.140@tcp20 [8/256/0/180] [15905.473819] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15905.533435] LNet: Added LNI 192.168.202.140@tcp21 [8/256/0/180] [15910.817658] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15910.840752] LNet: Added LNI 192.168.202.140@tcp22 [8/256/0/180] [15916.219829] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15916.246367] LNet: Added LNI 192.168.202.140@tcp23 [8/256/0/180] [15923.176242] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15923.213960] LNet: Added LNI 192.168.202.140@tcp24 [8/256/0/180] [15929.825362] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15929.874223] LNet: Added LNI 192.168.202.140@tcp25 [8/256/0/180] [15936.164616] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15936.195173] LNet: Added LNI 192.168.202.140@tcp26 [8/256/0/180] [15942.915153] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15942.955340] LNet: Added LNI 192.168.202.140@tcp27 [8/256/0/180] [15949.157179] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15949.198661] LNet: Added LNI 192.168.202.140@tcp28 [8/256/0/180] [15956.461954] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15956.512731] LNet: Added LNI 192.168.202.140@tcp29 [8/256/0/180] [15963.538214] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15963.568199] LNet: Added LNI 192.168.202.140@tcp30 [8/256/0/180] [15971.755966] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15971.870074] LNet: Added LNI 192.168.202.140@tcp31 [8/256/0/180] [15979.874581] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15979.894526] LNet: Added LNI 192.168.202.140@tcp32 [8/256/0/180] [15987.053192] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15987.079838] LNet: Added LNI 192.168.202.140@tcp33 [8/256/0/180] [15996.436774] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15996.466428] LNet: Added LNI 192.168.202.140@tcp34 [8/256/0/180] [16005.757791] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16005.809139] LNet: Added LNI 192.168.202.140@tcp35 [8/256/0/180] [16013.929918] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16013.951468] LNet: Added LNI 192.168.202.140@tcp36 [8/256/0/180] [16021.832408] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16021.885812] LNet: Added LNI 192.168.202.140@tcp37 [8/256/0/180] [16030.480030] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16030.517498] LNet: Added LNI 192.168.202.140@tcp38 [8/256/0/180] [16037.782326] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16037.810745] LNet: Added LNI 192.168.202.140@tcp39 [8/256/0/180] [16045.137989] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16045.159632] LNet: Added LNI 192.168.202.140@tcp40 [8/256/0/180] [16051.819858] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16051.841624] LNet: Added LNI 192.168.202.140@tcp41 [8/256/0/180] [16058.070734] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16058.102119] LNet: Added LNI 192.168.202.140@tcp42 [8/256/0/180] [16069.884599] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16069.928304] LNet: Added LNI 192.168.202.140@tcp43 [8/256/0/180] [16081.751405] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16081.777908] LNet: Added LNI 192.168.202.140@tcp44 [8/256/0/180] [16091.892082] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16091.917181] LNet: Added LNI 192.168.202.140@tcp45 [8/256/0/180] [16098.960915] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16098.981574] LNet: Added LNI 192.168.202.140@tcp46 [8/256/0/180] [16108.229710] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16108.254712] LNet: Added LNI 192.168.202.140@tcp47 [8/256/0/180] [16118.104156] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16118.142553] LNet: Added LNI 192.168.202.140@tcp48 [8/256/0/180] [16126.460203] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16126.523375] LNet: Added LNI 192.168.202.140@tcp49 [8/256/0/180] [16137.265652] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16137.292900] LNet: Added LNI 192.168.202.140@tcp50 [8/256/0/180] [16162.281985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16162.300344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16162.315973] Lustre: Skipped 1 previous similar message [16162.328285] Lustre: Skipped 2 previous similar messages [16165.899955] Lustre: server umount lustre-MDT0000 complete [16170.253499] LustreError: 473431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454134 with bad export cookie 334425967220939801 [16170.260307] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16170.267545] LustreError: 473431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16170.617868] Lustre: server umount lustre-MDT0001 complete [16188.896395] Lustre: 472710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454136/real 1788454136] req@0000000052f56c38 x1875329636550656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16188.923252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16188.945852] Lustre: Skipped 1 previous similar message [16189.848132] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [16190.948500] Lustre: 472709:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454139/real 1788454139] req@00000000b6e44f75 x1875329636550784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16192.995820] Lustre: 472708:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454141/real 1788454141] req@000000000ab7e735 x1875329636550912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16196.128096] Lustre: 472710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454144/real 1788454144] req@0000000024dbe36f x1875329636551168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454160 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16199.038630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16201.312176] Lustre: 472710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454149/real 1788454149] req@00000000b64fa1fc x1875329636551296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454165 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16215.584626] LustreError: 472707:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004e90f459 x1875329636552064/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16215.898871] LustreError: 475399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16215.926263] LustreError: 475399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16215.991516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16216.995407] LustreError: 475400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16217.001470] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16220.016134] LustreError: 475399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16220.845586] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16225.250771] LustreError: 475400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16230.383463] LustreError: 475399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16230.482031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16230.793588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16234.931081] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16235.803994] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16245.880984] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16251.685720] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16275.939728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16275.971809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16275.980757] Lustre: Skipped 1 previous similar message [16279.672236] Lustre: server umount lustre-OST0000 complete [16286.178447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16286.188129] Lustre: Skipped 1 previous similar message [16286.198573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16286.214245] Lustre: Skipped 1 previous similar message [16290.526091] Lustre: server umount lustre-MDT0000 complete [16291.299230] LustreError: 501237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16291.351042] LustreError: 501237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16295.058261] LustreError: 473430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454259 with bad export cookie 334425967220942426 [16295.060579] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16295.086603] LustreError: 473430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16295.424891] Lustre: server umount lustre-MDT0001 complete [16308.674667] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [16312.255414] Key type lgssc unregistered [16312.701703] LNet: 502454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16312.710545] LNetError: 502454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16312.718133] LNet: Removed LNI 192.168.202.140@tcp [16312.720604] LNet: Removed LNI 192.168.202.140@tcp10 [16312.729259] LNet: Removed LNI 192.168.202.140@tcp11 [16312.734743] LNet: Removed LNI 192.168.202.140@tcp12 [16312.742226] LNet: Removed LNI 192.168.202.140@tcp13 [16312.748470] LNet: Removed LNI 192.168.202.140@tcp14 [16312.754858] LNet: Removed LNI 192.168.202.140@tcp15 [16312.757591] LNet: Removed LNI 192.168.202.140@tcp16 [16312.761296] LNet: Removed LNI 192.168.202.140@tcp17 [16312.763798] LNet: Removed LNI 192.168.202.140@tcp18 [16312.770664] LNet: Removed LNI 192.168.202.140@tcp19 [16312.775538] LNet: Removed LNI 192.168.202.140@tcp20 [16312.778905] LNet: Removed LNI 192.168.202.140@tcp21 [16312.783734] LNet: Removed LNI 192.168.202.140@tcp22 [16312.789580] LNet: Removed LNI 192.168.202.140@tcp23 [16312.800921] LNet: Removed LNI 192.168.202.140@tcp24 [16312.804078] LNet: Removed LNI 192.168.202.140@tcp25 [16312.809482] LNet: Removed LNI 192.168.202.140@tcp26 [16312.811265] LNet: Removed LNI 192.168.202.140@tcp27 [16312.827167] LNet: Removed LNI 192.168.202.140@tcp28 [16312.833585] LNet: Removed LNI 192.168.202.140@tcp29 [16312.837782] LNet: Removed LNI 192.168.202.140@tcp30 [16312.840045] LNet: Removed LNI 192.168.202.140@tcp31 [16312.841375] LNet: Removed LNI 192.168.202.140@tcp32 [16312.848696] LNet: Removed LNI 192.168.202.140@tcp33 [16312.852847] LNet: Removed LNI 192.168.202.140@tcp34 [16312.854611] LNet: Removed LNI 192.168.202.140@tcp35 [16312.861673] LNet: Removed LNI 192.168.202.140@tcp36 [16312.866280] LNet: Removed LNI 192.168.202.140@tcp37 [16312.869247] LNet: Removed LNI 192.168.202.140@tcp38 [16312.871308] LNet: Removed LNI 192.168.202.140@tcp39 [16312.873826] LNet: Removed LNI 192.168.202.140@tcp40 [16312.875659] LNet: Removed LNI 192.168.202.140@tcp41 [16312.881032] LNet: Removed LNI 192.168.202.140@tcp42 [16312.883691] LNet: Removed LNI 192.168.202.140@tcp43 [16312.885393] LNet: Removed LNI 192.168.202.140@tcp44 [16312.895741] LNet: Removed LNI 192.168.202.140@tcp45 [16312.902646] LNet: Removed LNI 192.168.202.140@tcp46 [16312.908512] LNet: Removed LNI 192.168.202.140@tcp47 [16312.916965] LNet: Removed LNI 192.168.202.140@tcp48 [16312.924077] LNet: Removed LNI 192.168.202.140@tcp49 [16312.934799] LNet: Removed LNI 192.168.202.140@tcp50 [16313.797839] Key type .llcrypt unregistered [16313.799841] Key type ._llcrypt unregistered [16336.779183] Key type ._llcrypt registered [16336.781495] Key type .llcrypt registered [16336.876985] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [16353.362298] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [16354.647468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16354.768197] alg: No test for adler32 (adler32-zlib) [16355.875395] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16356.126385] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [16357.832794] Key type lgssc registered [16359.203496] Lustre: Echo OBD driver; http://www.lustre.org/ [16397.351246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16397.388687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16398.698248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16398.724572] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16398.794470] Lustre: lustre-MDT0000: new disk, initializing [16398.872689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16398.890846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16404.233558] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16417.266886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16417.492959] Lustre: 505651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16417.572688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16417.579157] Lustre: Skipped 1 previous similar message [16417.806533] Lustre: lustre-MDT0001: new disk, initializing [16418.006465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16418.037609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16418.048187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16425.067862] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16437.197872] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16447.127265] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16457.119540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16457.392652] Lustre: *** cfs_fail_loc=90f, val=0*** [16457.543843] Lustre: lustre-OST0000: new disk, initializing [16457.549065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16457.556297] Lustre: 507668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16457.652861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16465.139458] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16466.523554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16466.530887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16466.658287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16476.242615] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16487.399526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16487.400123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16487.412401] LustreError: Skipped 1 previous similar message [16487.413474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16487.428134] Lustre: Skipped 1 previous similar message [16487.440706] Lustre: Skipped 1 previous similar message [16491.085349] Lustre: server umount lustre-OST0000 complete [16497.642407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16497.659447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16497.659706] Lustre: Skipped 1 previous similar message [16497.671785] Lustre: Skipped 1 previous similar message [16501.765482] Lustre: server umount lustre-MDT0000 complete [16505.992117] LustreError: 505643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454470 with bad export cookie 762685011602743357 [16506.007152] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16506.503109] Lustre: server umount lustre-MDT0001 complete [16518.986804] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [16522.281261] Key type lgssc unregistered [16522.639780] LNet: 509573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16522.651855] LNetError: 509573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16522.681689] LNet: Removed LNI 192.168.202.140@tcp [16523.879237] Key type .llcrypt unregistered [16523.881209] Key type ._llcrypt unregistered [16548.781861] Key type ._llcrypt registered [16548.784763] Key type .llcrypt registered [16548.943223] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [16550.841588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16551.083920] alg: No test for adler32 (adler32-zlib) [16552.471255] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16552.649711] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [16554.337450] Key type lgssc registered [16556.724736] Lustre: Echo OBD driver; http://www.lustre.org/ [16568.228455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16568.254436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16569.879981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16574.639215] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16579.043124] LustreError: 510838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16579.055311] LustreError: 510838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16584.176527] LustreError: 510839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16584.640676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16585.363912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16590.797385] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16599.701313] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16605.674832] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16613.076292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16613.518599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16620.718792] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16628.488172] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16642.293584] loop0: detected capacity change from 0 to 204800000 [16642.487880] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16642.519665] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16642.538679] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16649.287353] loop0: detected capacity change from 0 to 204800000 [16649.535826] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16649.544895] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16649.554744] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16655.536332] loop0: detected capacity change from 0 to 204800000 [16661.893013] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16662.100099] Lustre: fs15246-OST0000: new disk, initializing [16662.108826] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16662.119649] Lustre: 514433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16662.199670] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16670.142393] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16682.919682] loop1: detected capacity change from 0 to 204800000 [16690.828504] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16690.977766] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16691.062319] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16691.237416] Lustre: fs15246-MDT0000: new disk, initializing [16691.408077] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16691.446734] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16691.613465] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16692.647347] Lustre: 515557:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000042fbf12e x1875330482620544/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788454672 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16696.625921] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16697.763610] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16697.771532] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16697.806745] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16722.915344] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16722.930195] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16728.044775] LustreError: 510833:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16728.501779] Lustre: server umount fs15246-MDT0000 complete [16733.158420] LustreError: 513330:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16738.280614] LustreError: 510835:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16739.858932] Lustre: server umount fs15246-OST0000 complete [16753.125706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16753.145207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16753.175479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16757.806995] Lustre: server umount lustre-OST0000 complete [16763.875077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16763.914288] Lustre: Skipped 1 previous similar message [16763.918759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16763.922611] Lustre: Skipped 1 previous similar message [16768.994678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16773.091329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16773.106083] Lustre: Skipped 2 previous similar messages [16776.160566] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16777.073842] Lustre: server umount lustre-MDT0000 complete [16778.210217] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16778.234972] LustreError: 510834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16782.433620] LustreError: 510820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454746 with bad export cookie 16517178004813367173 [16782.445707] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16782.447065] LustreError: 510820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16782.740762] Lustre: server umount lustre-MDT0001 complete [16796.301079] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [16798.603988] Key type lgssc unregistered [16798.924102] LNet: 518656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16798.941878] LNetError: 518656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16798.977219] LNet: Removed LNI 192.168.202.140@tcp [16800.419819] Key type .llcrypt unregistered [16800.425173] Key type ._llcrypt unregistered [16823.136940] Key type ._llcrypt registered [16823.140262] Key type .llcrypt registered [16823.373367] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [16824.918040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16825.018436] alg: No test for adler32 (adler32-zlib) [16826.196969] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16826.592111] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [16828.368238] Key type lgssc registered [16829.866083] Lustre: Echo OBD driver; http://www.lustre.org/ [16875.164905] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [16887.194584] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [16928.670386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16928.748711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16930.142171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16930.172438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16930.260752] Lustre: lustre-MDT0000: new disk, initializing [16930.363947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16930.379973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16936.328838] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16950.473438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16950.691653] Lustre: 523130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16950.791415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16950.800144] Lustre: Skipped 1 previous similar message [16950.923354] Lustre: lustre-MDT0001: new disk, initializing [16951.089025] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16951.127485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16951.152495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16957.379843] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [16970.873236] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16978.718104] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16987.200733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16987.571774] Lustre: lustre-OST0000: new disk, initializing [16987.579678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16987.588529] Lustre: 525106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16987.674565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16994.027205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16994.042920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16994.225189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16995.696253] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug -1 all [17006.824553] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17012.653348] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17012.929821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17020.637870] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17021.068906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17025.009125] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17025.013072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17025.044018] LustreError: Skipped 1 previous similar message [17025.047975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17025.050671] Lustre: Skipped 1 previous similar message [17029.717517] Lustre: server umount lustre-OST0000 complete [17035.233867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17035.240872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17035.256959] Lustre: Skipped 2 previous similar messages [17040.166201] Lustre: server umount lustre-MDT0000 complete [17044.497233] LustreError: 523124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455008 with bad export cookie 14252885685474049452 [17044.500468] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17044.507786] LustreError: 523124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17044.738067] Lustre: server umount lustre-MDT0001 complete [17079.667041] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [17091.494106] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [17136.836651] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16797 sec ========= 13:04:58 (1788455098) [17140.005937] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:05:01 (1788455101) === [17145.661945] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:05:07 (1788455107) === [17193.076725] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [17196.591584] Key type lgssc unregistered [17197.130746] LNet: 531757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17197.143577] LNetError: 531757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17197.164641] LNet: Removed LNI 192.168.202.140@tcp [17198.197329] Key type .llcrypt unregistered [17198.199424] Key type ._llcrypt unregistered