[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496842642 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002356] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007566] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.021429] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026167] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029005] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030125] *** VALIDATE tmpfs *** [ 0.032083] *** VALIDATE proc *** [ 0.033246] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.036190] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.043661] debug: unmapping init [mem 0xffffffff87459000-0xffffffff87460fff] [ 0.045174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046770] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050008] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054306] rcu: Hierarchical SRCU implementation. [ 0.056279] smp: Bringing up secondary CPUs ... [ 0.057571] x86: Booting SMP configuration: [ 0.058031] .... node #0, CPUs: #1 #2 #3 [ 0.064180] smp: Brought up 1 node, 4 CPUs [ 0.066025] smpboot: Max logical packages: 1 [ 0.067011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149441] node 0 deferred pages initialised in 80ms [ 0.153100] devtmpfs: initialized [ 0.154272] x86/mm: Memory block size: 128MB [ 0.158183] gcov: version magic: 0x41383552 [ 0.162030] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168365] pinctrl core: initialized pinctrl subsystem [ 0.170183] [ 0.170551] ************************************************************* [ 0.172011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174011] ** ** [ 0.176011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178013] ** ** [ 0.179007] ** This means that this kernel is built to expose internal ** [ 0.180014] ** IOMMU data structures, which may compromise security on ** [ 0.183012] ** your system. ** [ 0.185010] ** ** [ 0.187013] ** If you see this message and you are not debugging the ** [ 0.189014] ** kernel, report this immediately to your vendor! ** [ 0.191012] ** ** [ 0.193011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195011] ************************************************************* [ 0.197683] NET: Registered protocol family 16 [ 0.199526] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207016] cpuidle: using governor menu [ 0.208650] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211698] PCI: Using configuration type 1 for base access [ 0.214143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225225] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231314] cryptd: max_cpu_qlen set to 1000 [ 0.236202] ACPI: Added _OSI(Module Device) [ 0.237014] ACPI: Added _OSI(Processor Device) [ 0.239011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246437] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253362] ACPI: Interpreter enabled [ 0.254066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256016] ACPI: Using IOAPIC for interrupt routing [ 0.257115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260468] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270540] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281396] acpiphp: Slot [2] registered [ 0.283141] acpiphp: Slot [5] registered [ 0.285137] acpiphp: Slot [6] registered [ 0.286109] acpiphp: Slot [7] registered [ 0.287092] acpiphp: Slot [8] registered [ 0.289183] acpiphp: Slot [9] registered [ 0.290102] acpiphp: Slot [10] registered [ 0.292110] acpiphp: Slot [3] registered [ 0.293087] acpiphp: Slot [4] registered [ 0.295184] acpiphp: Slot [11] registered [ 0.296105] acpiphp: Slot [12] registered [ 0.297093] acpiphp: Slot [13] registered [ 0.299099] acpiphp: Slot [14] registered [ 0.300109] acpiphp: Slot [15] registered [ 0.302099] acpiphp: Slot [16] registered [ 0.303115] acpiphp: Slot [17] registered [ 0.305099] acpiphp: Slot [18] registered [ 0.306090] acpiphp: Slot [19] registered [ 0.307090] acpiphp: Slot [20] registered [ 0.308142] acpiphp: Slot [21] registered [ 0.310167] acpiphp: Slot [22] registered [ 0.311102] acpiphp: Slot [23] registered [ 0.313171] acpiphp: Slot [24] registered [ 0.314093] acpiphp: Slot [25] registered [ 0.315106] acpiphp: Slot [26] registered [ 0.318061] acpiphp: Slot [27] registered [ 0.319094] acpiphp: Slot [28] registered [ 0.320118] acpiphp: Slot [29] registered [ 0.321000] acpiphp: Slot [30] registered [ 0.321000] acpiphp: Slot [31] registered [ 0.323311] PCI host bridge to bus 0000:00 [ 0.325019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343215] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347333] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352805] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.366012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.372071] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.382023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384541] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.388199] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.390044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.398016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409983] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419585] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430126] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.444017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.466017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.476647] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.489016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.531623] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.543019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.555019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.584020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.594887] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.624023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.666603] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.719019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.739502] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.749017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.774018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.785122] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.787371] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.789339] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.795259] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799250] iommu: Default domain type: Passthrough [ 0.800551] SCSI subsystem initialized [ 0.801076] ACPI: bus type USB registered [ 0.802092] usbcore: registered new interface driver usbfs [ 0.804051] usbcore: registered new interface driver hub [ 0.805052] usbcore: registered new device driver usb [ 0.806119] pps_core: LinuxPPS API ver. 1 registered [ 0.807007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.809105] PTP clock support registered [ 0.811173] EDAC MC: Ver: 3.0.0 [ 0.813170] PCI: Using ACPI for IRQ routing [ 0.814617] NetLabel: Initializing [ 0.815008] NetLabel: domain hash size = 128 [ 0.816006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.817073] NetLabel: unlabeled traffic allowed by default [ 0.818115] vgaarb: loaded [ 0.819254] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.820010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.827289] clocksource: Switched to clocksource kvm-clock [ 0.947248] VFS: Disk quotas dquot_6.6.0 [ 0.948848] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.951358] *** VALIDATE ramfs *** [ 0.952610] *** VALIDATE hugetlbfs *** [ 0.954283] pnp: PnP ACPI init [ 0.956588] pnp: PnP ACPI: found 6 devices [ 0.990695] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.993951] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.996127] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.998236] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.000563] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.003069] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.005928] NET: Registered protocol family 2 [ 1.008495] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.013437] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.016274] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.020224] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.023155] TCP: Hash tables configured (established 65536 bind 65536) [ 1.026056] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.029048] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031958] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.034987] NET: Registered protocol family 1 [ 1.037396] RPC: Registered named UNIX socket transport module. [ 1.040230] RPC: Registered udp transport module. [ 1.041981] RPC: Registered tcp transport module. [ 1.044479] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.047583] NET: Registered protocol family 44 [ 1.049571] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.052179] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.056249] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.058770] PCI: CLS 0 bytes, default 64 [ 1.061298] Unpacking initramfs... [ 2.427478] debug: unmapping init [mem 0xffff9471bcc54000-0xffff9471bffbffff] [ 2.431832] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.433813] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.436682] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.923623] Initialise system trusted keyrings [ 2.925269] Key type blacklist registered [ 2.927037] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936277] zbud: loaded [ 2.939272] *** VALIDATE nfs *** [ 2.940517] *** VALIDATE nfs4 *** [ 2.942111] pstore: using deflate compression [ 2.945763] Platform Keyring initialized [ 3.040705] NET: Registered protocol family 38 [ 3.042544] Key type asymmetric registered [ 3.044115] Asymmetric key parser 'x509' registered [ 3.046449] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.050216] io scheduler mq-deadline registered [ 3.051644] io scheduler kyber registered [ 3.053726] io scheduler bfq registered [ 3.055334] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.058218] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.060398] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.062621] ACPI: Power Button [PWRF] [ 3.066320] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.072195] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.085931] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.090720] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.103260] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.130182] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.154773] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.158389] Non-volatile memory driver v1.3 [ 3.159972] Linux agpgart interface v0.103 [ 3.191179] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.193673] vda: detected capacity change from 0 to 76693504 [ 3.207379] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.210098] vdb: detected capacity change from 0 to 1073741824 [ 3.225179] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.227631] vdc: detected capacity change from 0 to 2621440000 [ 3.241373] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.243292] vdd: detected capacity change from 0 to 2621440000 [ 3.257704] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.259981] vde: detected capacity change from 0 to 4294967296 [ 3.271965] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.273675] vdf: detected capacity change from 0 to 4294967296 [ 3.278347] libphy: Fixed MDIO Bus: probed [ 3.284525] usbcore: registered new interface driver usbserial_generic [ 3.287500] usbserial: USB Serial support registered for generic [ 3.289708] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.294338] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.296105] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.297900] mousedev: PS/2 mouse device common for all mice [ 3.299633] rtc_cmos 00:05: RTC can wake from S4 [ 3.301386] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.306177] rtc_cmos 00:05: registered as rtc0 [ 3.307455] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.309991] intel_pstate: CPU model not supported [ 3.311359] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316641] hid: raw HID events driver (C) Jiri Kosina [ 3.320357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.320885] usbcore: registered new interface driver usbhid [ 3.326155] usbhid: USB HID core driver [ 3.327527] drop_monitor: Initializing network drop monitor service [ 3.329459] Initializing XFRM netlink socket [ 3.330749] NET: Registered protocol family 10 [ 3.333465] Segment Routing with IPv6 [ 3.334865] NET: Registered protocol family 17 [ 3.336373] mpls_gso: MPLS GSO support [ 3.340018] RAS: Correctable Errors collector initialized. [ 3.341767] AVX version of gcm_enc/dec engaged. [ 3.343295] AES CTR mode by8 optimization enabled [ 3.402370] sched_clock: Marking stable (3402350554, 0)->(4267978169, -865627615) [ 3.404754] registered taskstats version 1 [ 3.406074] Loading compiled-in X.509 certificates [ 3.407415] zswap: loaded using pool lzo/zbud [ 3.426390] Key type big_key registered [ 3.436331] Key type encrypted registered [ 3.437366] ima: No TPM chip found, activating TPM-bypass! [ 3.438527] ima: Allocated hash algorithm: sha1 [ 3.439694] ima: No architecture policies found [ 3.440833] evm: Initialising EVM extended attributes: [ 3.442078] evm: security.selinux [ 3.443134] evm: security.ima [ 3.444179] evm: security.capability [ 3.445598] evm: HMAC attrs: 0x1 [ 3.447333] rtc_cmos 00:05: setting system clock to 2026-09-05 00:30:40 UTC (1788568240) [ 3.451709] debug: unmapping init [mem 0xffffffff88403000-0xffffffff885fffff] [ 3.454526] debug: unmapping init [mem 0xffffffff87182000-0xffffffff87458fff] [ 3.462123] Write protecting the kernel read-only data: 28672k [ 3.464613] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.466541] debug: unmapping init [mem 0xffffffff86114000-0xffffffff861fffff] [ 3.494554] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.501848] systemd[1]: Detected virtualization kvm. [ 3.503580] systemd[1]: Detected architecture x86-64. [ 3.505211] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.532415] systemd[1]: No hostname configured. [ 3.533833] systemd[1]: Set hostname to . [ 3.535793] random: systemd: uninitialized urandom read (16 bytes read) [ 3.538284] systemd[1]: Initializing machine ID from random generator. [ 3.649046] random: systemd: uninitialized urandom read (16 bytes read) [ 3.650947] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.654171] random: systemd: uninitialized urandom read (16 bytes read) [ 3.655795] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.659173] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.165728] device-mapper: uevent: version 1.0.3 [ 4.168158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.904401] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.938469] random: fast init done [ 5.086239] scsi host0: ata_piix [ 5.132388] scsi host1: ata_piix [ 5.134374] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.137250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.894338] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.968159] random: crng init done [ 9.972637] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.088525] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.731495] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.053313] SELinux: Disabled at runtime. [ 17.301580] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.322357] systemd[1]: Detected virtualization kvm. [ 17.325813] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.827180] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.842814] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.864399] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.871556] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.874422] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.910057] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.936892] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. [ 20.328955] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.741566] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.907103] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.961397] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.445823] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.524640] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 29.577788] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 30.502910] NFS: Registering the id_resolver key type [ 30.507414] Key type id_resolver registered [ 30.508828] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Login Service. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg209-server login: [ 94.141248] libcfs: loading out-of-tree module taints kernel. [ 94.181556] Key type ._llcrypt registered [ 94.183616] Key type .llcrypt registered [ 94.326069] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 115.774575] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 117.806706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 117.820164] alg: No test for adler32 (adler32-zlib) [ 119.583678] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 120.797264] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 122.664168] Key type lgssc registered [ 124.961450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.366187] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 170.377050] hrtimer: interrupt took 5175115 ns [ 198.750355] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 214.575840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 214.617446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.944624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.964959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.068280] Lustre: lustre-MDT0000: new disk, initializing [ 216.194979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.216032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.629053] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.411664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.508394] Lustre: 6535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 237.531607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 237.535463] Lustre: Skipped 1 previous similar message [ 237.628850] Lustre: lustre-MDT0001: new disk, initializing [ 237.731362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.765562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.787268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 242.862462] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.550119] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 259.740502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 260.265584] Lustre: lustre-OST0000: new disk, initializing [ 260.278806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 260.296762] Lustre: 8475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 260.425424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.734320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 260.739847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 260.795576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 268.076768] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.426495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 284.634878] Lustre: lustre-OST0001: new disk, initializing [ 284.639832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 284.653132] Lustre: 9546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 284.777984] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 290.875656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 290.894651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 290.982928] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.125156] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.864671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 314.880533] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 321.955142] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing check_logdir /tmp/testlogs/ [ 328.236612] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing yml_node [ 333.635993] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 336.132557] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 339.593267] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 341.789645] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 4 20:36:17 EDT 2026 [ 366.054993] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 367.810573] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 386.529052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 386.534852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.546631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.067771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 388.072371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.097449] Lustre: Skipped 3 previous similar messages [ 389.983768] Lustre: server umount lustre-MDT0000 complete [ 395.196776] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568632 with bad export cookie 4271435739104667835 [ 395.222134] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 395.224024] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 398.305333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.309426] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.309450] LustreError: 6544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.309458] LustreError: 6544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 398.316768] Lustre: Skipped 3 previous similar messages [ 398.364922] Lustre: Skipped 1 previous similar message [ 400.358797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 400.358791] LustreError: 6544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.399412] LustreError: 6544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 402.001944] Lustre: server umount lustre-MDT0001 complete [ 406.376964] Lustre: server umount lustre-OST0000 complete [ 410.408329] Lustre: server umount lustre-OST0001 complete [ 417.817971] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 429.257498] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 472.182031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.594479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.673631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.808571] Lustre: lustre-MDT0000: new disk, initializing [ 473.015119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.056760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.685922] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 491.437561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.649469] Lustre: 15102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 491.732372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 491.738687] Lustre: Skipped 1 previous similar message [ 491.868606] Lustre: lustre-MDT0001: new disk, initializing [ 491.969238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 492.001201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 492.018207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 497.348511] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 507.896913] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 514.122571] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 521.481591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.865365] Lustre: lustre-OST0000: new disk, initializing [ 521.869427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 521.875610] Lustre: 17070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 521.982206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.016968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 523.027964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 523.232186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 530.696856] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 542.170472] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 547.368462] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 547.540572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 554.450614] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 554.736425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 559.087777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.097569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.117462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.117475] Lustre: Skipped 1 previous similar message [ 569.317113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.323516] Lustre: Skipped 4 previous similar messages [ 571.360266] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.680173] Lustre: server umount lustre-OST0000 complete [ 579.048593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.063787] LustreError: Skipped 1 previous similar message [ 579.073713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 579.094861] Lustre: Skipped 1 previous similar message [ 582.739590] Lustre: server umount lustre-MDT0000 complete [ 584.680027] LustreError: 15114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.701650] LustreError: 15114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 587.162914] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568824 with bad export cookie 4271435739104669368 [ 587.169038] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.178526] LustreError: 15095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 587.603894] Lustre: server umount lustre-MDT0001 complete [ 592.385068] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:40:28 (1788568828) [ 595.091994] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 597.211666] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:40:33 (1788568833) [ 605.454320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.843515] LustreError: 19486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 605.926376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.304259] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.606806] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 620.519819] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 620.546031] LustreError: 19487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 623.557809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.163992] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 641.530758] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 649.825483] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 658.191123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.709692] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 676.061717] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 685.545395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.563454] Lustre: Skipped 2 previous similar messages [ 685.563708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 685.564762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 685.588214] Lustre: Skipped 3 previous similar messages [ 692.162542] Lustre: server umount lustre-OST0000 complete [ 700.898275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 703.334965] Lustre: server umount lustre-MDT0000 complete [ 706.019351] LustreError: 19486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.044892] LustreError: 19486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 709.046051] LustreError: 19467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788568946 with bad export cookie 4271435739104670250 [ 709.060975] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.757645] Lustre: server umount lustre-MDT0001 complete [ 723.343939] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 727.001891] Key type lgssc unregistered [ 727.523982] LNet: 23023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.556500] LNetError: 23023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.579923] LNet: Removed LNI 192.168.202.109@tcp [ 729.084294] Key type .llcrypt unregistered [ 729.087721] Key type ._llcrypt unregistered [ 759.630250] Key type ._llcrypt registered [ 759.642898] Key type .llcrypt registered [ 759.778206] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 760.808082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 760.838331] alg: No test for adler32 (adler32-zlib) [ 762.344536] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 762.749499] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 764.498464] Key type lgssc registered [ 766.204723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 776.371406] loop: module loaded [ 776.475854] loop0: detected capacity change from 0 to 204800000 [ 776.776429] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 776.791861] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 777.170740] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 783.937271] loop0: detected capacity change from 0 to 204800000 [ 784.110985] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.127595] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 784.139513] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 790.750530] loop0: detected capacity change from 0 to 204800000 [ 797.083164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 797.142348] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.498898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 798.577602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 798.746750] Lustre: lustre-MDT0000: new disk, initializing [ 798.896138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 798.929130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 804.736090] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 815.864248] loop1: detected capacity change from 0 to 204800000 [ 822.019430] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 822.371809] Lustre: lustre-OST07c6: new disk, initializing [ 822.378827] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 822.383414] Lustre: Skipped 1 previous similar message [ 822.388795] Lustre: 26217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 822.460578] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 829.131839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 829.146332] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 829.181453] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 829.259520] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 834.421915] Lustre: 26733:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 834.433508] Lustre: 26733:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 834.439431] Lustre: 26733:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 834.448325] Lustre: 26733:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 834.456028] Lustre: 26733:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 834.461679] Lustre: 26733:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 848.488447] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 848.495031] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 848.507130] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.514130] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 848.520613] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 848.527422] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 853.660691] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 853.666579] Lustre: 25205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 853.671103] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 853.676029] Lustre: 25205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.683398] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 853.688947] Lustre: 25205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.696265] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 853.703435] Lustre: 25205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.708741] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 853.713348] Lustre: 25205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 853.718823] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 853.724833] Lustre: 25205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.042618] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 857.050161] Lustre: 25206:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 857.054082] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 857.058598] Lustre: 25206:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.068147] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 857.077026] Lustre: 25206:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.085825] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 857.098195] Lustre: 25206:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.108369] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 857.119424] Lustre: 25206:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.123958] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 857.133141] Lustre: 25206:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 862.304411] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 862.319183] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 862.327249] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 862.341143] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 862.351432] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 862.359159] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 862.370325] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 862.382078] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 862.391846] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 862.398181] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 862.406144] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 862.412769] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 870.424386] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 870.435119] Lustre: 25204:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 870.442318] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 870.450969] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 870.457969] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 870.468622] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 870.477959] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 870.488027] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 870.494139] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 870.500299] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 870.506379] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 870.513569] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 926.689401] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 926.727771] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 931.815356] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 936.931288] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 937.953920] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 938.172853] Lustre: server umount lustre-OST07c6 complete [ 950.216531] Lustre: server umount lustre-MDT0000 complete [ 972.144053] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 976.996820] Key type lgssc unregistered [ 977.271350] LNet: 28777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.301079] LNetError: 28777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.344611] LNet: Removed LNI 192.168.202.109@tcp [ 979.547616] Key type .llcrypt unregistered [ 979.558583] Key type ._llcrypt unregistered [ 1003.034661] Key type ._llcrypt registered [ 1003.036859] Key type .llcrypt registered [ 1003.159828] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1004.641303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1004.748845] alg: No test for adler32 (adler32-zlib) [ 1006.005503] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1006.201636] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1007.888207] Key type lgssc registered [ 1009.446899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1019.457455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1019.468943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.084984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1026.876732] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1031.649988] LustreError: 30051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.673984] LustreError: 30051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1036.770341] LustreError: 30052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.215586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.614235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1045.907841] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1055.066412] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.322590] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1071.013627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.317335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.515283] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1086.480396] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1091.908790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1110.500150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1110.503450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.527445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.547555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1112.549787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.593642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1115.930683] Lustre: server umount lustre-OST0000 complete [ 1121.765168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1121.781941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.797106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.807131] Lustre: Skipped 1 previous similar message [ 1126.936797] Lustre: server umount lustre-MDT0000 complete [ 1127.909789] LustreError: 30780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.929240] LustreError: 30780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1131.758970] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569368 with bad export cookie 11331359032116755923 [ 1131.766655] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.771491] LustreError: 30032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1132.389787] Lustre: server umount lustre-MDT0001 complete [ 1146.975399] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1150.589780] Key type lgssc unregistered [ 1151.009749] LNet: 33790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.021689] LNetError: 33790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.035241] LNet: Removed LNI 192.168.202.109@tcp [ 1152.050354] Key type .llcrypt unregistered [ 1152.065063] Key type ._llcrypt unregistered [ 1173.349794] Key type ._llcrypt registered [ 1173.356194] Key type .llcrypt registered [ 1173.474230] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1174.751343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.802020] alg: No test for adler32 (adler32-zlib) [ 1176.151470] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1176.531781] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1178.200190] Key type lgssc registered [ 1179.791393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1189.062605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1189.090298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.782778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.768731] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1200.103398] LustreError: 35060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1200.128390] LustreError: 35060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1205.223319] LustreError: 35061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.630910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.069787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1212.487931] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1220.672928] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1226.910804] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1234.423542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1234.753914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.765130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1242.085130] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1250.697678] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1266.661868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1266.672195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.687474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1267.686854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1267.687136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1267.712373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.167356] Lustre: server umount lustre-OST0000 complete [ 1277.925790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.949214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1277.957246] Lustre: Skipped 2 previous similar messages [ 1280.953829] Lustre: server umount lustre-MDT0000 complete [ 1283.046169] LustreError: 36040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1283.077043] LustreError: 36040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1285.230122] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569522 with bad export cookie 15742506697221187172 [ 1285.230582] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.238079] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1285.611933] Lustre: server umount lustre-MDT0001 complete [ 1297.820685] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1301.675494] Key type lgssc unregistered [ 1302.051995] LNet: 38614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.076881] LNetError: 38614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.099417] LNet: Removed LNI 192.168.202.109@tcp [ 1303.316451] Key type .llcrypt unregistered [ 1303.319843] Key type ._llcrypt unregistered [ 1328.664817] Key type ._llcrypt registered [ 1328.666434] Key type .llcrypt registered [ 1328.765475] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1329.961057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1330.188682] alg: No test for adler32 (adler32-zlib) [ 1331.317635] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1331.585542] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1333.280237] Key type lgssc registered [ 1334.713613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1342.624370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1342.637488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.229285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1348.741471] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1354.732649] LustreError: 40028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.758729] LustreError: 40028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1358.834598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1359.397783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.157458] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1374.443866] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1381.110359] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1388.517806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.052485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.168219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1396.752627] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1404.886424] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.832058] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1414.632969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1414.649844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.669407] Lustre: Skipped 1 previous similar message [ 1417.881786] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1417.894171] Lustre: Skipped 1 previous similar message [ 1418.745045] Lustre: server umount lustre-MDT0000 complete [ 1419.760789] LustreError: 42485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.776283] LustreError: 42485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1422.843283] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569659 with bad export cookie 17656765270815844760 [ 1422.855066] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1422.865812] LustreError: 40008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1422.992228] LustreError: 42485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.020903] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 1424.872300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.894638] Lustre: Skipped 2 previous similar messages [ 1427.873198] LustreError: 42485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.878138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1427.887962] LustreError: 42485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1427.909136] Lustre: Skipped 1 previous similar message [ 1429.698996] Lustre: server umount lustre-MDT0001 complete [ 1451.840605] Lustre: server umount lustre-OST0000 complete [ 1471.169685] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1474.249214] Key type lgssc unregistered [ 1474.531388] LNet: 43892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.536541] LNetError: 43892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.572422] LNet: Removed LNI 192.168.202.109@tcp [ 1475.538382] Key type .llcrypt unregistered [ 1475.539823] Key type ._llcrypt unregistered [ 1501.321902] Key type ._llcrypt registered [ 1501.324425] Key type .llcrypt registered [ 1501.426584] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1502.793402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.823054] alg: No test for adler32 (adler32-zlib) [ 1503.909791] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1504.272662] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1505.960435] Key type lgssc registered [ 1507.418672] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1515.629116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1515.656087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.133667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.193365] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1527.276734] LustreError: 45307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.311726] LustreError: 45307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.647099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.077803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1535.808905] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1544.552695] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1550.663819] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1558.451678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.980228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.108059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1566.555488] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1574.390568] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1581.539055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1581.562519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.567261] Lustre: Skipped 1 previous similar message [ 1582.249933] Lustre: lustre-OST0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1586.661060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1587.839696] Lustre: server umount lustre-OST0000 complete [ 1605.526231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1605.556815] Lustre: Skipped 1 previous similar message [ 1605.562922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.569864] Lustre: Skipped 3 previous similar messages [ 1610.721070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1610.726785] Lustre: Skipped 1 previous similar message [ 1611.978550] Lustre: server umount lustre-MDT0000 complete [ 1615.665191] LustreError: 45289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788569852 with bad export cookie 2918360891060485374 [ 1615.676162] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1615.690179] LustreError: 45289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1622.088640] Lustre: server umount lustre-MDT0001 complete [ 1634.473880] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1638.468789] Key type lgssc unregistered [ 1639.136676] LNet: 49018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.140972] LNetError: 49018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.173924] LNet: Removed LNI 192.168.202.109@tcp [ 1641.539724] Key type .llcrypt unregistered [ 1641.540882] Key type ._llcrypt unregistered [ 1668.670741] Key type ._llcrypt registered [ 1668.672789] Key type .llcrypt registered [ 1668.806223] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1669.726774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1669.751759] alg: No test for adler32 (adler32-zlib) [ 1670.861333] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1671.162576] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1672.824198] Key type lgssc registered [ 1673.925800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1681.474394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1681.487752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.959538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.712929] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1692.131267] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.150247] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1697.249512] LustreError: 50436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.760749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1698.252254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1703.027528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1710.989895] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1717.185439] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1723.779665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1723.823931] LustreError: 52258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.049233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.192607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1730.727957] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1738.467784] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.733339] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1748.792169] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1761.838210] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0420h59m56s [ 1763.914464] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1768.421272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.427363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.442383] Lustre: Skipped 1 previous similar message [ 1768.480187] Lustre: Skipped 2 previous similar messages [ 1769.508340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.514227] Lustre: Skipped 2 previous similar messages [ 1771.172811] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1772.535373] Lustre: server umount lustre-MDT0000 complete [ 1775.586530] LustreError: 51437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.607219] LustreError: 51437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1776.385329] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1781.732587] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.753624] LustreError: 50435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1783.265216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.428734] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1783.777240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.851929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1784.825752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1789.160410] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1793.253919] Lustre: DEBUG MARKER: Wait for df (36720) ... [ 1795.538863] Lustre: DEBUG MARKER: done [ 1800.173072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.184622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.189514] Lustre: Skipped 2 previous similar messages [ 1800.196341] Lustre: Skipped 2 previous similar messages [ 1805.285182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1805.287904] Lustre: Skipped 1 previous similar message [ 1805.771525] Lustre: server umount lustre-OST0000 complete [ 1810.402299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.403278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1810.410030] Lustre: Skipped 1 previous similar message [ 1815.669400] Lustre: server umount lustre-MDT0000 complete [ 1816.481404] LustreError: 51437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.496028] LustreError: 51437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1819.377756] LustreError: 50416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570056 with bad export cookie 17883813784694503027 [ 1819.387255] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.627414] Lustre: server umount lustre-MDT0001 complete [ 1832.572368] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1835.765294] Key type lgssc unregistered [ 1836.099280] LNet: 55373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.105727] LNetError: 55373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.169443] LNet: Removed LNI 192.168.202.109@tcp [ 1838.449924] Key type .llcrypt unregistered [ 1838.451501] Key type ._llcrypt unregistered [ 1881.513887] Key type ._llcrypt registered [ 1881.519696] Key type .llcrypt registered [ 1881.649319] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1882.756252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1882.843154] alg: No test for adler32 (adler32-zlib) [ 1884.158919] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 1884.366400] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1886.088187] Key type lgssc registered [ 1887.331262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1894.510416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1894.526585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.870835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1895.901352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1896.047569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.263900] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1910.949424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.041957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1911.065345] Lustre: 57579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1911.378113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1916.712354] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1924.759958] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1931.443708] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1938.933519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1939.150229] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1939.480371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1945.527197] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1946.259271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1954.065710] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1960.651456] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0421h03m16s [ 1962.610393] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1970.355914] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1972.818680] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1973.124115] LustreError: 60058:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1993.458594] Lustre: lustre-MDT0000: Client 16cd72ac-cd5d-4aad-97c3-7a653e5c8d87 (at 192.168.202.9@tcp) reconnecting [ 2013.152458] LustreError: 60058:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2015.122746] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2017.835788] Lustre: DEBUG MARKER: done [ 2023.397846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2023.401402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.405151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2023.436683] Lustre: Skipped 1 previous similar message [ 2025.418847] Lustre: server umount lustre-OST0000 complete [ 2033.635930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.638389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.665342] Lustre: Skipped 1 previous similar message [ 2033.680493] Lustre: Skipped 1 previous similar message [ 2034.724799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2034.729224] Lustre: Skipped 2 previous similar messages [ 2036.016967] Lustre: server umount lustre-MDT0000 complete [ 2039.946663] LustreError: 57569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570276 with bad export cookie 4146260706697754457 [ 2039.954733] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.229787] Lustre: server umount lustre-MDT0001 complete [ 2054.219301] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2057.616685] Key type lgssc unregistered [ 2058.135727] LNet: 61907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.150634] LNetError: 61907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.180193] LNet: Removed LNI 192.168.202.109@tcp [ 2059.434079] Key type .llcrypt unregistered [ 2059.435929] Key type ._llcrypt unregistered [ 2106.153868] Key type ._llcrypt registered [ 2106.155964] Key type .llcrypt registered [ 2106.266199] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2107.621569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.665082] alg: No test for adler32 (adler32-zlib) [ 2108.731610] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2109.049467] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2110.745917] Key type lgssc registered [ 2112.065973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.387585] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2115.406208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.706724] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2116.745716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2116.957169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2117.847570] Lustre: Failing over lustre-MDT0000 [ 2118.204560] Lustre: server umount lustre-MDT0000 complete [ 2125.205838] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:06:01 (1788570361) [ 2131.959521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2132.471749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.123678] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2147.050510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.122754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2147.147830] Lustre: 64702:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2147.341732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.423778] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2161.182966] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.277782] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2175.215439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2175.475368] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2175.706788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.256345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2182.244240] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2191.069738] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2197.365921] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2207.203408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.211317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2207.220488] Lustre: Skipped 2 previous similar messages [ 2207.241575] Lustre: Skipped 2 previous similar messages [ 2212.707086] Lustre: server umount lustre-MDT0000 complete [ 2216.787515] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570453 with bad export cookie 4172127134386027221 [ 2216.793080] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.798045] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2217.176987] Lustre: server umount lustre-MDT0001 complete [ 2220.926705] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2225.644125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2233.825793] Lustre: 63625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570454/real 1788570454] req@000000008cbf7516 x1875451939168896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2233.853620] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.288437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2236.960154] Lustre: 63625:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570458/real 1788570458] req@00000000b8fc7b1d x1875451939169024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570474 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.944353] Lustre: 63623:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570459/real 1788570459] req@00000000e7686ac4 x1875451939169152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570475 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2241.057527] LustreError: 63621:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001a3c4633 x1875451939169920/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2241.405093] LustreError: 66548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2241.422519] LustreError: 66548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2241.505706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.604755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2242.465090] Lustre: 63622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570463/real 1788570463] req@00000000fbce9a73 x1875451939169408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.567815] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.584765] LustreError: 68467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2246.442940] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2247.655137] Lustre: 63624:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570468/real 1788570468] req@0000000089dc5858 x1875451939169536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2248.675729] LustreError: 68469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.697555] LustreError: 68469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2253.800283] LustreError: 68468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.828308] LustreError: 68468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2255.339780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2255.700941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2260.554579] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2260.974513] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2262.007546] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2269.689971] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2276.962942] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2286.563909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2286.571444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.585669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.586047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.605701] Lustre: Skipped 1 previous similar message [ 2290.861234] Lustre: server umount lustre-MDT0000 complete [ 2292.707682] LustreError: 68469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.727520] LustreError: 68469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2295.416199] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570532 with bad export cookie 4172127134386029503 [ 2295.420268] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2295.428040] LustreError: 64692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2295.977824] Lustre: server umount lustre-MDT0001 complete [ 2303.242350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2313.184353] Lustre: 63622:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788570534/real 1788570534] req@00000000721378e5 x1875451939198464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788570550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2313.218663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.248943] Lustre: Skipped 2 previous similar messages [ 2314.300088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.441235] LustreError: 63621:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000721378e5 x1875451939199488/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2321.846352] LustreError: 66549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.940639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.004563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2327.448172] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2337.377839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.153590] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2343.218509] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2352.328881] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2359.337234] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2367.985748] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.999344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2368.005919] Lustre: Skipped 1 previous similar message [ 2373.092849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.106282] Lustre: Skipped 4 previous similar messages [ 2381.280144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2381.667922] Lustre: server umount lustre-MDT0000 complete [ 2383.332445] LustreError: 71154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.347131] LustreError: 71154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2386.118956] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570623 with bad export cookie 4172127134386030882 [ 2386.121047] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2386.140205] LustreError: 66564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2386.576339] Lustre: server umount lustre-MDT0001 complete [ 2394.303711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2398.305493] Lustre: DEBUG MARKER: files compared the same [ 2410.580515] Lustre: server umount lustre-OST0000 complete [ 2427.429699] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2430.737552] Key type lgssc unregistered [ 2431.095297] LNet: 74429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.111419] LNetError: 74429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.134218] LNet: Removed LNI 192.168.202.109@tcp [ 2432.244219] Key type .llcrypt unregistered [ 2432.246511] Key type ._llcrypt unregistered [ 2460.040059] Key type ._llcrypt registered [ 2460.042534] Key type .llcrypt registered [ 2460.194550] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2461.666506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2461.814968] alg: No test for adler32 (adler32-zlib) [ 2463.211968] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2463.610919] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2465.338800] Key type lgssc registered [ 2466.714836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2475.886762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.906348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.438756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2482.190252] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug +malloc all [ 2487.780755] LustreError: 75843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.803123] LustreError: 75843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2491.815956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.076919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.784732] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug +malloc all [ 2505.551985] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2511.901519] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2519.960867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2520.370501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.412485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2527.406220] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug +malloc all [ 2536.924815] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2548.195190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2548.197489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2548.214030] LustreError: Skipped 1 previous similar message [ 2548.220878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2548.245836] Lustre: Skipped 1 previous similar message [ 2548.284274] Lustre: Skipped 1 previous similar message [ 2553.320040] LustreError: 77663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.351133] LustreError: 77663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2553.438515] Lustre: server umount lustre-OST0000 complete [ 2558.433277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.437609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.462560] Lustre: Skipped 1 previous similar message [ 2558.473534] Lustre: Skipped 1 previous similar message [ 2563.561084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.641022] Lustre: server umount lustre-MDT0000 complete [ 2568.687075] LustreError: 75843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.707328] LustreError: 75824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788570805 with bad export cookie 7396041523055140847 [ 2568.725384] LustreError: 75843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2568.737523] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.747756] LustreError: 75824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2569.331388] Lustre: server umount lustre-MDT0001 complete [ 2583.290487] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2586.678985] Key type lgssc unregistered [ 2586.964425] LNet: 79377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2586.972802] LNetError: 79377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.006891] LNet: Removed LNI 192.168.202.109@tcp [ 2589.275682] Key type .llcrypt unregistered [ 2589.280908] Key type ._llcrypt unregistered [ 2618.197349] Key type ._llcrypt registered [ 2618.202597] Key type .llcrypt registered [ 2618.343756] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2619.781564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2619.987708] alg: No test for adler32 (adler32-zlib) [ 2621.268132] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2621.658961] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2623.433522] Key type lgssc registered [ 2625.035769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2634.469385] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2634.495568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2650.209396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2655.328353] LustreError: 80766:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 2682.016835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.409589] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2701.343476] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2710.685303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.902597] Lustre: *** cfs_fail_loc=706, val=0*** [ 2711.537073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.644183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2716.375578] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2716.652768] LustreError: 80791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.672378] LustreError: 80791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2717.694451] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.718397] LustreError: 80792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2722.806962] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.821529] LustreError: 81855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2727.910341] LustreError: 82194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2727.935732] LustreError: 82194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2728.067512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.567316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2735.569985] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2745.965344] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2753.580665] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2757.604302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2757.606640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.628196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.140097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.152732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2759.167092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2762.493873] Lustre: server umount lustre-OST0000 complete [ 2769.381846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.391891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.418449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.426562] Lustre: Skipped 1 previous similar message [ 2774.116615] Lustre: server umount lustre-MDT0000 complete [ 2775.015709] LustreError: 81854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.036699] LustreError: 81854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2778.955114] LustreError: 82595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571016 with bad export cookie 8872270578756791151 [ 2778.963360] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2778.973086] LustreError: 82595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2779.418450] Lustre: server umount lustre-MDT0001 complete [ 2793.186446] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2797.124300] Key type lgssc unregistered [ 2797.473986] LNet: 84385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2797.479506] LNetError: 84385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2797.501670] LNet: Removed LNI 192.168.202.109@tcp [ 2798.684266] Key type .llcrypt unregistered [ 2798.689794] Key type ._llcrypt unregistered [ 2831.878948] Key type ._llcrypt registered [ 2831.880399] Key type .llcrypt registered [ 2831.976563] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2833.553445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2833.608025] alg: No test for adler32 (adler32-zlib) [ 2834.813277] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 2835.108608] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2837.024100] Key type lgssc registered [ 2838.842813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2852.495282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2852.532181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.038847] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2869.881845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.650200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2875.873828] LustreError: 86488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.893323] LustreError: 86488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2877.016714] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2879.986249] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.017634] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2885.093798] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.107865] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2888.064120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.655372] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.692362] LustreError: 86487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2888.723070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.806852] LustreError: 87239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.842144] LustreError: 87239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2897.466623] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2906.083978] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.096411] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2907.696921] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2907.718455] Lustre: MGS: Client 20665061-a34f-4b12-9f53-230111328df1 (at 0@lo) reconnecting [ 2907.768375] LustreError: 86502:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2908.046337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2908.139464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2913.655554] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2930.147792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2930.154985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.178671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2933.731804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2933.735835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2933.754379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2935.179306] Lustre: server umount lustre-OST0000 complete [ 2943.970710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2943.974928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.985253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.002580] Lustre: Skipped 1 previous similar message [ 2944.023069] Lustre: Skipped 2 previous similar messages [ 2945.723413] Lustre: server umount lustre-MDT0000 complete [ 2949.093260] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2949.116791] LustreError: 86491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2950.753394] Lustre: server umount lustre-MDT0001 complete [ 2955.155356] Lustre: server umount MGS complete [ 2968.382280] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:20:04 (1788571204) [ 3001.439085] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3015.286639] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3060.768631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.233631] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3078.657671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.033802] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3079.094546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3079.206308] Lustre: lustre-MDT0001: new disk, initializing [ 3079.307408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3079.319863] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3079.323800] Lustre: 93684:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3079.328364] LustreError: 93684:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3085.327804] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3097.867101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3098.162858] Lustre: lustre-OST0000: new disk, initializing [ 3098.167517] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3098.176174] Lustre: 94450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3098.262192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.379789] Lustre: 94746:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c8da30e8 x1875452700438400/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788571358 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3105.961969] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3117.560510] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3125.566283] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3125.574716] Lustre: MGS: Client c358eb22-66eb-45c6-9493-1f39963babbf (at 0@lo) reconnecting [ 3125.598738] Lustre: 92952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3125.613989] LustreError: 93670:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3125.659404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3125.729028] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3125.800516] Lustre: lustre-MDT0000: new disk, initializing [ 3126.026720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3126.059254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3127.141034] Lustre: 95520:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000976cda06 x1875452700445568/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788571380 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3131.464740] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3132.325216] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3136.240168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3136.247509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3136.381701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3161.059242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3161.061791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3161.076373] Lustre: Skipped 1 previous similar message [ 3161.102335] Lustre: Skipped 1 previous similar message [ 3164.433149] Lustre: server umount lustre-OST0000 complete [ 3171.299560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3171.300101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3171.308715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3171.350706] Lustre: Skipped 1 previous similar message [ 3174.820364] Lustre: server umount lustre-MDT0000 complete [ 3176.427232] LustreError: 93656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3176.436272] LustreError: 93656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3193.313895] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3193.711322] Lustre: server umount lustre-MDT0001 complete [ 3198.010618] Lustre: server umount MGS complete [ 3210.909233] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:24:06 (1788571446) [ 3231.950840] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3235.751163] Key type lgssc unregistered [ 3236.249534] LNet: 98017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3236.259574] LNetError: 98017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3236.301761] LNet: Removed LNI 192.168.202.109@tcp [ 3237.635712] Key type .llcrypt unregistered [ 3237.639231] Key type ._llcrypt unregistered [ 3260.339787] Key type ._llcrypt registered [ 3260.344717] Key type .llcrypt registered [ 3260.489732] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3262.432950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3262.553576] alg: No test for adler32 (adler32-zlib) [ 3263.618190] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3263.943928] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3265.704213] Key type lgssc registered [ 3267.616993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3277.870241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3277.885128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.451747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.501721] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3289.570959] LustreError: 99241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.590757] LustreError: 99241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3294.690196] LustreError: 99240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3296.495555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.896479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3302.906509] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3312.829649] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3320.565613] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3329.377606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3329.759970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3338.420996] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3348.301536] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3349.254956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3364.842754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.850817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.879926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.235535] Lustre: server umount lustre-OST0000 complete [ 3373.538364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3373.555569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.569636] Lustre: Skipped 1 previous similar message [ 3373.583323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.585933] Lustre: Skipped 1 previous similar message [ 3375.076626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3375.078247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.665417] Lustre: server umount lustre-MDT0000 complete [ 3380.205963] LustreError: 99235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3380.226317] LustreError: 99235:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3382.099080] LustreError: 99221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571619 with bad export cookie 1703407564504484385 [ 3382.120631] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3382.605412] Lustre: server umount lustre-MDT0001 complete [ 3398.365071] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3402.156430] Key type lgssc unregistered [ 3402.617079] LNet: 102795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3402.630875] LNetError: 102795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3402.661857] LNet: Removed LNI 192.168.202.109@tcp [ 3404.093480] Key type .llcrypt unregistered [ 3404.098721] Key type ._llcrypt unregistered [ 3427.915513] Key type ._llcrypt registered [ 3427.919397] Key type .llcrypt registered [ 3428.184417] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3429.634909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.655271] alg: No test for adler32 (adler32-zlib) [ 3431.048569] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3431.356958] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3433.194437] Key type lgssc registered [ 3434.217527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3441.943806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3441.955666] LustreError: 103871:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3441.970118] LustreError: 103875:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3441.976796] LustreError: 103871:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3441.987313] LustreError: 103871:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3441.995650] LustreError: 103871:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3442.000826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.760092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.225727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.623147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.909390] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3466.423755] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing lsmod [ 3470.693410] LustreError: 105081:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3470.714770] LustreError: 105082:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3470.723742] LustreError: 105081:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3470.734350] LustreError: 105081:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3470.748202] LustreError: 105081:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3470.768543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.461369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.451771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.466936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.474927] Lustre: Skipped 1 previous similar message [ 3476.498915] Lustre: Skipped 2 previous similar messages [ 3478.567179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3478.590195] Lustre: Skipped 2 previous similar messages [ 3481.443783] Lustre: server umount lustre-MDT0000 complete [ 3483.627737] LustreError: 104407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3483.666552] LustreError: 104407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3485.525209] LustreError: 103887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571722 with bad export cookie 184696989349065061 [ 3485.538287] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3486.820190] LustreError: 104343:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b9aa5e07 x1875453325366656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3487.098967] Lustre: server umount lustre-MDT0001 complete [ 3501.818921] Lustre: server umount lustre-OST0000 complete [ 3515.110642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.092559] LustreError: 106147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.271221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.513319] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3522.661621] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3526.626254] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3531.746441] LustreError: 106147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.746678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3534.285660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3539.154205] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3549.235675] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3558.122503] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.542894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.901892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3572.875867] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3582.371523] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3586.377800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3595.750280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3595.756961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.771293] Lustre: Skipped 1 previous similar message [ 3595.777548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.795141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.825216] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3600.887681] Lustre: server umount lustre-OST0000 complete [ 3605.985691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3606.002493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.039261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3606.048590] Lustre: Skipped 2 previous similar messages [ 3611.845816] Lustre: server umount lustre-MDT0000 complete [ 3612.130657] LustreError: 107145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3615.756491] LustreError: 106129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788571852 with bad export cookie 184696989349065761 [ 3615.773911] LustreError: 106129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3615.795922] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3616.063881] Lustre: server umount lustre-MDT0001 complete [ 3630.025836] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3633.211484] Key type lgssc unregistered [ 3633.638671] LNet: 109683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3633.651183] LNetError: 109683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3633.671351] LNet: Removed LNI 192.168.202.109@tcp [ 3634.818461] Key type .llcrypt unregistered [ 3634.822570] Key type ._llcrypt unregistered [ 3665.929566] Key type ._llcrypt registered [ 3665.931939] Key type .llcrypt registered [ 3666.138179] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3667.770920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3667.900930] alg: No test for adler32 (adler32-zlib) [ 3669.164570] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3669.435133] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3671.160174] Key type lgssc registered [ 3672.783787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3682.756517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3682.785755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.388656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.119525] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3694.570921] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.584453] LustreError: 111100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3699.693733] LustreError: 111099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3700.921027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3701.586953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3707.632487] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3717.280766] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3725.466367] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3733.060963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3733.448728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.073770] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3750.167351] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3754.942102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3759.957084] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3764.197664] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3770.342169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3770.348816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.365695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.469799] LustreError: 112922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.495172] LustreError: 112922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3775.592965] Lustre: server umount lustre-OST0000 complete [ 3780.579318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.589317] Lustre: Skipped 1 previous similar message [ 3780.605216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.617476] Lustre: Skipped 2 previous similar messages [ 3785.705507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.716138] Lustre: Skipped 1 previous similar message [ 3790.817984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.832620] Lustre: Skipped 1 previous similar message [ 3793.888149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3794.405625] Lustre: server umount lustre-MDT0000 complete [ 3795.942105] LustreError: 111099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.965357] LustreError: 111099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3797.912903] LustreError: 111079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572034 with bad export cookie 6518361874241860662 [ 3797.919664] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3797.927936] LustreError: 111079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3798.316856] Lustre: server umount lustre-MDT0001 complete [ 3813.147405] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3815.745912] Key type lgssc unregistered [ 3816.074866] LNet: 114742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.081463] LNetError: 114742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3816.109970] LNet: Removed LNI 192.168.202.109@tcp [ 3817.815360] Key type .llcrypt unregistered [ 3817.818266] Key type ._llcrypt unregistered [ 3839.697394] Key type ._llcrypt registered [ 3839.698899] Key type .llcrypt registered [ 3839.859551] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3841.921880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3842.021649] alg: No test for adler32 (adler32-zlib) [ 3843.158956] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 3843.576463] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3845.336158] Key type lgssc registered [ 3846.604760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3855.713955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3855.744591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.458994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3863.229631] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3867.618987] LustreError: 115967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3867.647965] LustreError: 115967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3872.740200] LustreError: 115967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.044661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3874.490551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.582777] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3889.844336] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3897.375466] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3905.416398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3905.871326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.849261] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3924.506494] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3927.414701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3939.809586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3939.814359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.825833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.881019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3942.886888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3942.896115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3942.926071] Lustre: Skipped 1 previous similar message [ 3944.997490] Lustre: server umount lustre-OST0000 complete [ 3951.074510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3951.080343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.090286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.124579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3953.127557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.247108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3958.254063] Lustre: Skipped 2 previous similar messages [ 3963.366988] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3963.573086] Lustre: server umount lustre-MDT0000 complete [ 3967.461680] LustreError: 115947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572204 with bad export cookie 10202939620266850633 [ 3967.476423] LustreError: 115947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3967.478902] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3967.939946] Lustre: server umount lustre-MDT0001 complete [ 3980.604146] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3984.607944] Key type lgssc unregistered [ 3984.934812] LNet: 119525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3984.952372] LNetError: 119525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3984.988065] LNet: Removed LNI 192.168.202.109@tcp [ 3986.244611] Key type .llcrypt unregistered [ 3986.247139] Key type ._llcrypt unregistered [ 4017.187460] Key type ._llcrypt registered [ 4017.192111] Key type .llcrypt registered [ 4017.348472] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4019.505261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4019.707305] alg: No test for adler32 (adler32-zlib) [ 4021.013886] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4021.432768] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4023.208171] Key type lgssc registered [ 4024.665909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4033.229643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4033.259497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.824578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.718145] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4044.265244] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4044.281594] LustreError: 120955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4047.910283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.701956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4056.279516] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4066.750454] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4074.711815] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4083.402446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.936916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.213911] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4101.837442] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4104.458257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4115.199086] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4120.215878] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4120.227099] Lustre: Skipped 2 previous similar messages [ 4123.767046] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4123.802413] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4126.741241] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4126.750971] Lustre: Skipped 2 previous similar messages [ 4130.395765] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4130.405071] Lustre: Skipped 1 previous similar message [ 4137.442943] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4137.450539] Lustre: Skipped 2 previous similar messages [ 4140.143698] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.9@tcp 192.168.0.[2,10]@tcp [ 4151.947350] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.9/24@tcp in log lustre-MDT0000 [ 4151.958896] Lustre: Skipped 2 previous similar messages [ 4157.553689] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.9/24@tcp [ 4157.562890] Lustre: Skipped 1 previous similar message [ 4170.216259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.225926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.242174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.239760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.240703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.252388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4173.036527] Lustre: server umount lustre-OST0000 complete [ 4181.474404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.478616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.497818] Lustre: Skipped 1 previous similar message [ 4181.512101] Lustre: Skipped 2 previous similar messages [ 4184.026635] Lustre: server umount lustre-MDT0000 complete [ 4186.593600] LustreError: 121871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.613283] LustreError: 121871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4187.929862] LustreError: 120935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572424 with bad export cookie 11037377449006622986 [ 4187.938169] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4187.941064] LustreError: 120935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4188.530647] Lustre: server umount lustre-MDT0001 complete [ 4203.065310] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4206.922750] Key type lgssc unregistered [ 4207.347710] LNet: 125808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4207.353368] LNetError: 125808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4207.368757] LNet: Removed LNI 192.168.202.109@tcp [ 4208.622134] Key type .llcrypt unregistered [ 4208.624640] Key type ._llcrypt unregistered [ 4246.572650] Key type ._llcrypt registered [ 4246.574750] Key type .llcrypt registered [ 4246.687918] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4247.967984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4248.192036] alg: No test for adler32 (adler32-zlib) [ 4249.478442] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4249.807514] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4251.536495] Key type lgssc registered [ 4253.266847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4262.805939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4262.830409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.522841] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4264.536872] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.9/24@tcp [ 4264.578558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4269.883474] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4274.165702] LustreError: 127371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.207753] LustreError: 127371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4279.270490] LustreError: 127370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4280.831284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.148582] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4281.193126] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.9/24@tcp [ 4281.225724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4286.817889] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4295.919289] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4303.229799] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4312.035072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4312.542673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.225696] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4330.465551] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4334.244995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4347.364335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4347.379624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4347.395684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.929287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4349.931119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4349.934382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.749076] Lustre: server umount lustre-OST0000 complete [ 4358.113528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4358.116200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.141435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.143786] Lustre: Skipped 1 previous similar message [ 4360.162176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4360.162610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.032381] Lustre: server umount lustre-MDT0000 complete [ 4364.296233] LustreError: 127350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572601 with bad export cookie 7672151262122611620 [ 4364.300305] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.300501] LustreError: 127350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4364.428811] Lustre: server umount lustre-MDT0001 complete [ 4368.205870] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4369.281589] Key type lgssc unregistered [ 4369.410246] LNet: 130966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4369.412434] LNetError: 130966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4369.419319] LNet: Removed LNI 192.168.202.109@tcp [ 4369.713725] Key type .llcrypt unregistered [ 4369.714772] Key type ._llcrypt unregistered [ 4377.633925] Key type ._llcrypt registered [ 4377.635160] Key type .llcrypt registered [ 4377.673354] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4383.456601] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4383.846094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4383.874662] alg: No test for adler32 (adler32-zlib) [ 4384.751823] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4384.857026] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4386.440154] Key type lgssc registered [ 4386.880992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4398.452579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4398.460593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.554627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4399.565471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4399.605628] Lustre: lustre-MDT0000: new disk, initializing [ 4399.628280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.635615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4401.023426] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4405.184496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.234914] Lustre: 134332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4405.267576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4405.270945] Lustre: Skipped 1 previous similar message [ 4405.319571] Lustre: lustre-MDT0001: new disk, initializing [ 4405.360639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.377902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4405.382583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4406.814338] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4410.668814] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4412.655800] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4415.023551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4415.132508] Lustre: lustre-OST0000: new disk, initializing [ 4415.134156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4415.137343] Lustre: 136301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4415.167928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4417.210036] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4420.790364] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4422.136188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4422.139848] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4422.175649] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4422.212273] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4422.295382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4423.665714] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4423.738868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4427.235041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.237330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4427.238141] LustreError: Skipped 1 previous similar message [ 4427.241866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4427.244199] Lustre: Skipped 1 previous similar message [ 4430.491487] Lustre: server umount lustre-OST0000 complete [ 4432.353725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.354511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.357235] Lustre: Skipped 2 previous similar messages [ 4437.473606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4437.475613] Lustre: Skipped 2 previous similar messages [ 4442.593628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4442.595757] Lustre: Skipped 1 previous similar message [ 4446.176103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4446.243237] Lustre: server umount lustre-MDT0000 complete [ 4447.552713] LustreError: 135127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572684 with bad export cookie 368100195752657350 [ 4447.555461] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.559204] LustreError: 135127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4447.674708] Lustre: server umount lustre-MDT0001 complete [ 4450.628908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.816645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.279664] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4455.261179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.405591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4456.913428] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4459.354123] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4461.265971] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4463.619443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4463.728239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.731800] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4468.222903] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4469.761169] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4475.949104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 4477.441311] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4477.526717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4481.000298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4481.063474] Lustre: lustre-OST0001: new disk, initializing [ 4481.065124] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4481.067525] Lustre: 141927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4481.098520] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4483.116596] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4486.492839] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4489.217473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4489.220627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4489.248737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4489.652546] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4491.111348] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4491.212719] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4493.055698] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff954982a48000.ost_server_uuid 50 [ 4493.572100] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff954982a48000.ost_server_uuid in FULL state after 0 sec [ 4499.430433] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4499.430596] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4499.434797] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4499.436309] LustreError: Skipped 1 previous similar message [ 4499.439413] Lustre: Skipped 1 previous similar message [ 4501.569319] Lustre: server umount lustre-OST0001 complete [ 4504.549379] LustreError: 140337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4504.549701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.555791] LustreError: 140337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4504.561631] Lustre: Skipped 2 previous similar messages [ 4509.669908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4509.670860] LustreError: 140337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.671909] Lustre: Skipped 3 previous similar messages [ 4509.678652] LustreError: 140337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4514.790191] LustreError: 140336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.795466] LustreError: 140336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4517.344150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4517.393299] Lustre: server umount lustre-OST0000 complete [ 4519.905643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.909250] Lustre: Skipped 1 previous similar message [ 4524.843060] Lustre: server umount lustre-MDT0000 complete [ 4525.026038] LustreError: 138517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4525.031282] LustreError: 138517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4526.109450] LustreError: 140357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572763 with bad export cookie 368100195752658232 [ 4526.112923] LustreError: 140357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4526.113239] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4526.253882] Lustre: server umount lustre-MDT0001 complete [ 4532.925987] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4534.122782] Key type lgssc unregistered [ 4534.263558] LNet: 144945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4534.266137] LNetError: 144945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4534.272478] LNet: Removed LNI 192.168.202.109@tcp [ 4534.612541] Key type .llcrypt unregistered [ 4534.614376] Key type ._llcrypt unregistered [ 4547.652814] Key type ._llcrypt registered [ 4547.653835] Key type .llcrypt registered [ 4547.695342] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4554.071996] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4554.447932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4554.470230] alg: No test for adler32 (adler32-zlib) [ 4555.348916] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4555.443453] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4557.024147] Key type lgssc registered [ 4557.444159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4568.400338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4568.406306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.498883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.508873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.550063] Lustre: lustre-MDT0000: new disk, initializing [ 4569.571256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.577313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4570.966916] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4575.190996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.240964] Lustre: 149065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4575.267254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4575.270140] Lustre: Skipped 1 previous similar message [ 4575.317279] Lustre: lustre-MDT0001: new disk, initializing [ 4575.354551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4575.367330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4575.370469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4576.785311] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4580.525629] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4582.357431] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4584.599950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4584.723296] Lustre: lustre-OST0000: new disk, initializing [ 4584.725160] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4584.728403] Lustre: 151035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4584.756568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4586.715072] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4588.610088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4588.614668] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4588.655370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4590.508701] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4592.487929] Lustre: Failing over lustre-OST0000 [ 4592.552378] Lustre: server umount lustre-OST0000 complete [ 4605.426678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4605.551696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.559154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.568513] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4606.864623] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4607.528668] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4608.802954] Lustre: Failing over lustre-MDT0000 [ 4608.908681] Lustre: server umount lustre-MDT0000 complete [ 4609.824189] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572830/real 1788572830] req@000000009cfb0fcb x1875454503969408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788572846 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4610.656223] Lustre: 147294:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572831/real 1788572831] req@000000005c9ffd67 x1875454503970304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788572847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4611.043806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.047342] Lustre: Skipped 2 previous similar messages [ 4615.136143] Lustre: 147295:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572836/real 1788572836] req@000000000d48c13b x1875454503971456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788572852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4616.161218] LustreError: 149877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.166025] LustreError: 149877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4617.352749] LustreError: 151956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.192082] Lustre: 147296:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572841/real 1788572841] req@00000000f6e1ccc3 x1875454503972608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788572857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.282083] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.286644] LustreError: 149077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4621.887326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.927751] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.931505] Lustre: 149070:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4622.020832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4622.041827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4622.079613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4622.472867] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4623.416625] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4627.431183] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4627.439292] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4627.462355] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4632.547954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4632.548759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4632.554358] Lustre: Skipped 1 previous similar message [ 4632.556536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4632.558368] Lustre: Skipped 1 previous similar message [ 4634.640508] Lustre: server umount lustre-OST0000 complete [ 4637.666051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4637.666151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4637.667064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4637.669918] Lustre: Skipped 2 previous similar messages [ 4637.673262] LustreError: Skipped 1 previous similar message [ 4642.074736] Lustre: server umount lustre-MDT0000 complete [ 4642.785749] LustreError: 149877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.791801] LustreError: 149877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4643.346490] LustreError: 149059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788572880 with bad export cookie 6093226866032654814 [ 4643.350320] LustreError: 149059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4643.350534] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4643.475802] Lustre: server umount lustre-MDT0001 complete [ 4647.020854] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4648.017599] Key type lgssc unregistered [ 4648.136358] LNet: 154678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4648.138232] LNetError: 154678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4648.144237] LNet: Removed LNI 192.168.202.109@tcp [ 4648.435275] Key type .llcrypt unregistered [ 4648.436272] Key type ._llcrypt unregistered [ 4668.080148] Key type ._llcrypt registered [ 4668.081468] Key type .llcrypt registered [ 4668.121328] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4668.472355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4668.507893] alg: No test for adler32 (adler32-zlib) [ 4669.362287] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 4669.457574] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4671.032187] Key type lgssc registered [ 4671.379867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4673.928757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4673.936445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.100687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.443344] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4680.168814] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4682.177028] LustreError: 156220:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4682.180551] LustreError: 156220:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4682.183732] LustreError: 156220:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 4682.311864] Lustre: server umount lustre-MDT0000 complete [ 4687.273502] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4690.524303] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4702.586114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.680723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4702.692090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4702.733140] Lustre: lustre-MDT0000: new disk, initializing [ 4702.754892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.760435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4704.180678] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4708.372487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4708.416494] Lustre: 159517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4708.440497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4708.442289] Lustre: Skipped 1 previous similar message [ 4708.480249] Lustre: lustre-MDT0001: new disk, initializing [ 4708.513781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4708.526873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4708.531059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4709.890252] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4713.571187] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4715.353079] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4717.588141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4717.702554] Lustre: lustre-OST0000: new disk, initializing [ 4717.704480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4717.707332] Lustre: 161484:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4717.740947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4719.552359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4719.556138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4719.598895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4719.688242] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4723.366226] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4726.218080] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4726.778852] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:49:24 (1788572964) [ 4790.752621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4790.755316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4790.759301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.265699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.265824] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.268774] Lustre: Skipped 1 previous similar message [ 4791.273787] Lustre: Skipped 1 previous similar message [ 4796.330649] Lustre: server umount lustre-MDT0000 complete [ 4796.385614] LustreError: 159526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.391048] LustreError: 159526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4797.576553] LustreError: 159507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573034 with bad export cookie 6768574988110480957 [ 4797.577365] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4797.582795] LustreError: 159507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4801.504605] LustreError: 162201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4801.504633] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.513559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4802.528459] LustreError: 159522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.745624] Lustre: server umount lustre-MDT0001 complete [ 4811.155830] Lustre: server umount lustre-OST0000 complete [ 4814.239945] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4817.487502] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4829.401985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.497471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4829.509241] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4829.549740] Lustre: lustre-MDT0000: new disk, initializing [ 4829.575460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.583384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4830.942406] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4835.005415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.050595] Lustre: 166110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4835.095886] LustreError: 166121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.103063] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4835.105049] Lustre: Skipped 1 previous similar message [ 4835.147055] Lustre: lustre-MDT0001: new disk, initializing [ 4835.199647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4835.204399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4836.551662] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4840.379990] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4842.207978] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4844.522163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4844.635994] Lustre: lustre-OST0000: new disk, initializing [ 4844.638254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4844.642198] Lustre: 168081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4844.672313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4844.675758] Lustre: Skipped 1 previous similar message [ 4846.629992] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4850.171728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4850.176171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4850.211921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4850.450946] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4851.920309] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4851.994112] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4853.405402] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4853.495792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4855.266791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4855.267223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.268948] LustreError: Skipped 1 previous similar message [ 4855.271083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4855.273191] Lustre: Skipped 1 previous similar message [ 4855.274427] Lustre: Skipped 1 previous similar message [ 4856.281439] Lustre: server umount lustre-OST0000 complete [ 4860.385941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.386358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.389739] Lustre: Skipped 1 previous similar message [ 4860.392873] Lustre: Skipped 2 previous similar messages [ 4863.641658] Lustre: server umount lustre-MDT0000 complete [ 4864.884539] LustreError: 166102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573101 with bad export cookie 6768574988110639787 [ 4864.889693] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.892623] LustreError: 166102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4865.023411] Lustre: server umount lustre-MDT0001 complete [ 4867.905644] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:51:45 (1788573105) [ 4873.207739] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4876.160652] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4887.762489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.852274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4887.861385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4887.900929] Lustre: lustre-MDT0000: new disk, initializing [ 4887.920513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.925967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4889.248561] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4893.492243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.542993] Lustre: 172830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4893.651372] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4895.077744] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4898.845180] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4900.713521] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4902.973948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4903.089970] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4903.093042] Lustre: 174807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4905.121948] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4908.532093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4908.540457] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4908.845266] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4912.115130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.889933] Lustre: server umount lustre-OST0000 complete [ 4919.265612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4919.269499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4919.271357] Lustre: Skipped 2 previous similar messages [ 4924.386312] Lustre: server umount lustre-MDT0000 complete [ 4925.677118] LustreError: 172868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573162 with bad export cookie 6768574988110640669 [ 4925.679187] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.682337] LustreError: 172868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4925.798988] Lustre: server umount lustre-MDT0001 complete [ 4928.467408] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:52:45 (1788573165) [ 4933.845401] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4936.844508] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4948.475132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.568393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4948.578655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4948.580478] Lustre: Skipped 3 previous similar messages [ 4948.620138] Lustre: lustre-MDT0000: new disk, initializing [ 4948.621506] Lustre: Skipped 2 previous similar messages [ 4948.643365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.645539] Lustre: Skipped 2 previous similar messages [ 4948.650881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4948.653440] Lustre: Skipped 2 previous similar messages [ 4950.066879] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4954.195854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4954.237605] Lustre: 179115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4954.352352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4955.690842] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4959.451402] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4961.338700] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4963.656455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4963.778562] Lustre: 181091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4964.639147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4965.871534] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4969.654848] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4975.075150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4975.076087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4975.077731] LustreError: Skipped 1 previous similar message [ 4975.081151] Lustre: Skipped 2 previous similar messages [ 4975.083509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4975.086071] Lustre: Skipped 3 previous similar messages [ 4977.621152] Lustre: server umount lustre-OST0000 complete [ 4980.194647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4985.053261] Lustre: server umount lustre-MDT0000 complete [ 4985.313132] LustreError: 179127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4985.318690] LustreError: 179127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4986.301371] LustreError: 179110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573223 with bad export cookie 6768574988110641726 [ 4986.304606] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4986.304692] LustreError: 179110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4986.420596] Lustre: server umount lustre-MDT0001 complete [ 4989.983397] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4991.053095] Key type lgssc unregistered [ 4991.185378] LNet: 182943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4991.187661] LNetError: 182943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4991.198361] LNet: Removed LNI 192.168.202.109@tcp [ 4991.545001] Key type .llcrypt unregistered [ 4991.546810] Key type ._llcrypt unregistered [ 5000.653170] Key type ._llcrypt registered [ 5000.654221] Key type .llcrypt registered [ 5000.693346] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5001.068986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5001.123364] alg: No test for adler32 (adler32-zlib) [ 5001.990631] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5002.093435] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5003.680190] Key type lgssc registered [ 5004.056152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5006.756987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5006.763144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.931589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.390895] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5012.387154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5012.533276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.928478] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5016.290188] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5018.091022] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5020.332526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5020.450780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.403541] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5024.724470] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.767992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5048.289419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5048.292437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.296079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.314129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5049.314681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5049.316117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5052.749823] Lustre: server umount lustre-OST0000 complete [ 5054.433602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.434491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5054.437015] Lustre: Skipped 1 previous similar message [ 5054.438743] Lustre: Skipped 1 previous similar message [ 5059.553637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5059.555616] Lustre: Skipped 2 previous similar messages [ 5064.674090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5064.676241] Lustre: Skipped 1 previous similar message [ 5068.256079] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5068.315283] Lustre: server umount lustre-MDT0000 complete [ 5069.564367] LustreError: 184312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573306 with bad export cookie 5561324242191415155 [ 5069.567169] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5069.568649] LustreError: 184312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5069.678936] Lustre: server umount lustre-MDT0001 complete [ 5073.550823] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5074.622595] Key type lgssc unregistered [ 5074.746387] LNet: 187885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5074.748959] LNetError: 187885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5074.756289] LNet: Removed LNI 192.168.202.109@tcp [ 5075.075983] Key type .llcrypt unregistered [ 5075.077714] Key type ._llcrypt unregistered [ 5084.341248] Key type ._llcrypt registered [ 5084.342455] Key type .llcrypt registered [ 5084.381932] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5084.762686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5084.828152] alg: No test for adler32 (adler32-zlib) [ 5085.695292] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5085.793836] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5087.376087] Key type lgssc registered [ 5087.771132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5090.339051] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5090.345484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.493342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5092.853235] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5095.649351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.779939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5097.119447] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5099.471548] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5101.327916] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5103.569369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5103.680407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5105.527276] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5107.939659] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5117.030577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5121.505746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5121.508027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.511891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.530644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.531927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.532220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.538575] Lustre: Skipped 1 previous similar message [ 5123.727830] Lustre: lustre-OST0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 5127.649398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5127.651310] Lustre: Skipped 1 previous similar message [ 5132.771274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.773334] Lustre: Skipped 2 previous similar messages [ 5135.328151] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5135.384747] Lustre: server umount lustre-OST0000 complete [ 5137.829905] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff954982fd2800.ost_server_uuid 50 [ 5138.339911] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff954982fd2800.ost_server_uuid in DISCONN state after 0 sec [ 5138.885954] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5152.224920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5152.227570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5152.232106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5152.235281] Lustre: Skipped 1 previous similar message [ 5155.033510] Lustre: server umount lustre-MDT0000 complete [ 5156.326285] LustreError: 189256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573393 with bad export cookie 3969423021538887648 [ 5156.329491] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5156.330642] LustreError: 189256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5156.462957] Lustre: server umount lustre-MDT0001 complete [ 5159.259698] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:56:36 (1788573396) [ 5161.557463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.717216] LustreError: 193232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5161.722696] LustreError: 193232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5161.741261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5163.071537] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5165.943481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5166.064224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5167.450261] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5169.816456] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5171.588501] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5173.742691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5175.809029] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5178.248402] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5180.476090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5180.560666] Lustre: lustre-OST0001: new disk, initializing [ 5180.562618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5180.565077] Lustre: 196070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5180.600533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5180.602892] Lustre: Skipped 1 previous similar message [ 5182.149972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5182.153295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5182.199556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5182.593131] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5186.286389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5186.305678] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5187.554825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5187.557357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5187.561147] Lustre: Skipped 1 previous similar message [ 5187.563084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5187.565615] Lustre: Skipped 2 previous similar messages [ 5191.653128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5191.657542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5193.277698] Lustre: server umount lustre-OST0000 complete [ 5195.199834] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5195.269967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5196.697392] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5196.772292] LustreError: 195560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.776430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5196.778173] LustreError: 195560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5201.892585] LustreError: 195560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5201.896944] LustreError: 195560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5207.011370] LustreError: 196774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.012018] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5207.017809] LustreError: 196774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5207.022272] Lustre: Skipped 1 previous similar message [ 5212.134856] LustreError: 195051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.141056] LustreError: 195051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5212.881229] Lustre: server umount lustre-OST0001 complete [ 5217.252477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5220.262077] Lustre: server umount lustre-MDT0000 complete [ 5221.565209] LustreError: 193215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573458 with bad export cookie 3969423021538888705 [ 5221.569351] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.569363] LustreError: 193215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5221.690392] Lustre: server umount lustre-MDT0001 complete [ 5229.320382] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:57:46 (1788573466) [ 5231.450246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5231.546143] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5231.560504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5231.645473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.976851] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5235.660923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.696547] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5235.711662] Lustre: 199487:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5237.198704] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5239.566683] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5241.486702] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.883405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.950207] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5246.019620] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5246.991953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5248.446475] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5250.767179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5250.792995] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5252.809661] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5255.168454] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5257.185288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5257.188765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.193657] Lustre: Skipped 2 previous similar messages [ 5257.196048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.197961] Lustre: Skipped 10 previous similar messages [ 5262.141696] Lustre: server umount lustre-OST0000 complete [ 5263.331395] LustreError: 201332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5263.336154] LustreError: 201332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5278.861112] Lustre: server umount lustre-OST0001 complete [ 5286.171670] Lustre: server umount lustre-MDT0000 complete [ 5287.409345] LustreError: 199479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573524 with bad export cookie 3969423021538890091 [ 5287.412789] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5287.415835] LustreError: 199479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5287.530664] Lustre: server umount lustre-MDT0001 complete [ 5295.129864] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:58:52 (1788573532) [ 5300.204176] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 5303.118151] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5315.044558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.127310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5315.136700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5315.173483] Lustre: lustre-MDT0000: new disk, initializing [ 5315.193733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5315.195919] Lustre: Skipped 3 previous similar messages [ 5315.201054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5316.550724] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5320.732910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5320.776733] Lustre: 207543:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5320.803303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5320.805275] Lustre: Skipped 1 previous similar message [ 5320.841991] Lustre: lustre-MDT0001: new disk, initializing [ 5320.891586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5320.896550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5322.211156] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5325.887786] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5327.738099] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5329.919185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5330.024126] Lustre: lustre-OST0000: new disk, initializing [ 5330.025989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5330.028701] Lustre: 209510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5331.997735] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5333.239954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5333.243504] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5333.277429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5335.757944] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5337.228265] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5337.310375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5338.724852] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5338.806733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5343.713485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.715141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.716689] LustreError: Skipped 5 previous similar messages [ 5343.720408] Lustre: Skipped 5 previous similar messages [ 5343.721127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.725437] Lustre: Skipped 11 previous similar messages [ 5345.555034] Lustre: server umount lustre-OST0000 complete [ 5361.120102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5361.187139] Lustre: server umount lustre-MDT0000 complete [ 5362.421240] LustreError: 209509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573599 with bad export cookie 3969423021538891638 [ 5362.424454] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5362.424888] LustreError: 209509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5362.536237] Lustre: server umount lustre-MDT0001 complete [ 5365.283676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.427673] LustreError: 211724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.433193] LustreError: 211724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5366.760329] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5369.597776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5371.094107] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5373.439388] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5375.185893] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5377.397340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5379.437984] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5381.773312] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5383.161596] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5392.405740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 5393.741266] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5393.806173] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5400.529421] Lustre: server umount lustre-OST0000 complete [ 5402.524686] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5403.893239] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5427.440742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.550365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5427.553744] Lustre: Skipped 5 previous similar messages [ 5429.451430] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5431.770658] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5432.262384] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5435.279912] Lustre: server umount lustre-OST0000 complete [ 5436.545917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5436.549583] Lustre: Skipped 5 previous similar messages [ 5438.678798] Lustre: server umount lustre-MDT0000 complete [ 5439.903187] LustreError: 211705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573676 with bad export cookie 3969423021538892520 [ 5439.905612] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.907922] LustreError: 211705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5440.028997] Lustre: server umount lustre-MDT0001 complete [ 5442.804225] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:01:20 (1788573680) [ 5444.908668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.053833] LustreError: 217756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5446.396056] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5449.190255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.636564] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5452.982727] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5454.766155] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5456.967517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5459.068610] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5461.383619] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5462.730066] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5470.956467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 5472.276795] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5472.344291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5474.530694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.600353] Lustre: lustre-OST0001: new disk, initializing [ 5474.601929] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5474.604115] Lustre: 221177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.505416] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5480.170098] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5480.450409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5480.454175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5480.483458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5481.573133] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5481.643579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5482.989400] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5483.077829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5485.540113] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5485.543113] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5485.543131] LustreError: Skipped 3 previous similar messages [ 5485.545351] Lustre: Skipped 10 previous similar messages [ 5489.789518] Lustre: server umount lustre-OST0001 complete [ 5491.775533] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5491.859224] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5493.226096] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5493.309157] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5511.144048] LustreError: 223172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5511.148530] LustreError: 223172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5516.971391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5519.003479] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5521.500270] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5522.010213] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5522.408700] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5522.412862] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5522.413729] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5528.763470] Lustre: server umount lustre-OST0001 complete [ 5555.168082] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5555.221163] Lustre: server umount lustre-OST0000 complete [ 5562.782621] Lustre: server umount lustre-MDT0000 complete [ 5563.989570] LustreError: 217736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573801 with bad export cookie 3969423021538893451 [ 5563.992861] LustreError: 217736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5563.994579] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5570.168284] Lustre: server umount lustre-MDT0001 complete [ 5577.924316] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:03:35 (1788573815) [ 5580.110171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.199593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5580.209153] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5580.278798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5580.281378] Lustre: Skipped 5 previous similar messages [ 5581.595312] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5584.350055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.382362] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5584.392296] Lustre: 226416:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5585.809767] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5588.173325] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5589.993993] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5592.196705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.262418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5594.265376] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5596.674906] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5600.413987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5600.438765] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5602.455692] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5604.837353] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5606.360657] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5607.743394] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5608.835216] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5610.321644] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5610.392598] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5612.219779] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9549ab9ce000.ost_server_uuid 50 [ 5612.707313] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9549ab9ce000.ost_server_uuid in FULL state after 0 sec [ 5613.062126] Lustre: Permanently deactivating lustre-OST0001 [ 5613.068244] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5613.913753] Lustre: Permanently reactivating lustre-OST0001 [ 5618.658754] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5618.659822] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.661217] LustreError: Skipped 6 previous similar messages [ 5618.665178] Lustre: Skipped 8 previous similar messages [ 5620.924459] Lustre: server umount lustre-OST0001 complete [ 5628.303643] Lustre: server umount lustre-OST0000 complete [ 5635.671948] Lustre: server umount lustre-MDT0000 complete [ 5636.905290] LustreError: 226407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573873 with bad export cookie 3969423021538894725 [ 5636.907755] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.908615] LustreError: 226407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5637.017158] Lustre: server umount lustre-MDT0001 complete [ 5644.544489] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:04:41 (1788573881) [ 5646.955962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.052357] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5647.060925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5647.064044] Lustre: Skipped 5 previous similar messages [ 5648.460828] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5651.299747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.333829] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5651.343262] Lustre: 233104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5652.781777] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5655.110934] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5656.875500] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5659.070953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.146900] Lustre: Permanently deactivating lustre-OST0000 [ 5659.151287] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5661.092994] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5663.386902] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5665.272362] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5665.646128] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5667.733976] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5670.121245] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5672.469716] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5676.078410] Lustre: Permanently reactivating lustre-OST0000 [ 5676.083802] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5676.086065] Lustre: Skipped 2 previous similar messages [ 5681.649247] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5681.677164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5686.014316] Lustre: server umount lustre-OST0001 complete [ 5687.778944] LustreError: 234951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.784195] LustreError: 234951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 5693.325919] Lustre: server umount lustre-OST0000 complete [ 5700.894247] Lustre: server umount lustre-MDT0000 complete [ 5702.135353] LustreError: 233847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788573939 with bad export cookie 3969423021538896384 [ 5702.140765] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5702.269571] Lustre: server umount lustre-MDT0001 complete [ 5705.818885] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5706.807750] Key type lgssc unregistered [ 5706.935345] LNet: 237951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5706.937666] LNetError: 237951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5706.945315] LNet: Removed LNI 192.168.202.109@tcp [ 5707.283768] Key type .llcrypt unregistered [ 5707.284840] Key type ._llcrypt unregistered [ 5715.897221] Key type ._llcrypt registered [ 5715.898287] Key type .llcrypt registered [ 5715.936314] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5716.320297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.329861] alg: No test for adler32 (adler32-zlib) [ 5717.182692] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5717.269106] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5718.848186] Key type lgssc registered [ 5719.190462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5722.446723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5722.452652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.606556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.907183] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5727.689760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5727.726460] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5727.736122] Lustre: 239379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5727.741736] Lustre: Permanently deactivating lustre-MDT0001 [ 5727.747658] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5727.856082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5729.265583] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5731.742776] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5733.660366] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5733.867536] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5735.938717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.063963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.122718] LustreError: 241228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.130572] LustreError: 241228:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5737.999015] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5740.379849] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5742.563749] LustreError: 241231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.698150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5742.769739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5744.697568] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5747.066363] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5748.203476] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5749.225503] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5749.226048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5752.882761] Lustre: Permanently reactivating lustre-MDT0001 [ 5752.887933] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5752.889822] Lustre: Skipped 2 previous similar messages [ 5755.726896] Lustre: lustre-MDT0001: Client 8b3e7909-8ebe-4f32-bf21-8112ebde4ab0 (at 192.168.202.9@tcp) reconnecting [ 5757.881957] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5758.475171] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5759.495348] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.499871] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5759.502835] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5759.507019] LustreError: 243314:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5759.507599] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5759.510650] LustreError: 243314:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5759.517505] LustreError: 243314:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -108 [ 5759.521789] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5759.521789] [ 5759.524655] LustreError: 239381:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5760.237739] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5760.325989] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5761.692687] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5761.786810] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5762.649307] Lustre: Permanently deactivating lustre-MDT0001 [ 5762.656282] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5762.658159] Lustre: Skipped 2 previous similar messages [ 5766.468112] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5773.793410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5773.795818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.797599] LustreError: Skipped 1 previous similar message [ 5773.802162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5774.818606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5774.822751] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.826320] Lustre: Skipped 2 previous similar messages [ 5776.607887] Lustre: server umount lustre-MDT0000 complete [ 5777.842292] LustreError: 239369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574014 with bad export cookie 18284478777968950698 [ 5777.845335] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.847089] LustreError: 239369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5777.955659] Lustre: server umount lustre-MDT0001 complete [ 5785.277395] Lustre: server umount lustre-OST0000 complete [ 5788.909322] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:07:06 (1788574026) [ 5792.272649] Lustre: server umount lustre-OST0001 complete [ 5794.256796] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 5797.274144] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5808.682148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.785220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5808.790245] Lustre: Skipped 2 previous similar messages [ 5808.799635] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5808.838233] Lustre: lustre-MDT0000: new disk, initializing [ 5808.858513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.864840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5810.193137] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5814.344995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.384153] Lustre: 247682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5814.414945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5814.416128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5814.417063] Lustre: Skipped 3 previous similar messages [ 5814.418846] Lustre: Skipped 1 previous similar message [ 5814.458395] Lustre: lustre-MDT0001: new disk, initializing [ 5814.498930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5814.503098] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5815.821883] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5819.511962] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.276440] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5823.495543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.590090] Lustre: lustre-OST0000: new disk, initializing [ 5823.591725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5823.594027] Lustre: 249651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5823.623152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.624958] Lustre: Skipped 1 previous similar message [ 5825.534211] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5828.606269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5828.610249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5828.648031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5829.166319] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5829.774206] LustreError: 247717:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5833.993391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.071437] Lustre: lustre-OST0001: new disk, initializing [ 5834.072990] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5834.075266] Lustre: 250768:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5836.139447] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5838.345115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5838.349254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5838.390648] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5839.836468] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5839.864056] LustreError: 247717:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5839.872958] LustreError: 247688:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5840.333974] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5849.968149] LustreError: 247688:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5850.768620] Lustre: lustre-OST0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 5853.667266] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5853.669760] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5853.673575] Lustre: Skipped 1 previous similar message [ 5855.883162] Lustre: lustre-OST0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 5855.885658] Lustre: Skipped 2 previous similar messages [ 5856.704560] Lustre: server umount lustre-OST0001 complete [ 5858.787847] LustreError: 250322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.792631] LustreError: 250322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5863.906176] LustreError: 249636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.910831] LustreError: 249636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5869.027681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.027783] LustreError: 249634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5869.029173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.032162] Lustre: Skipped 1 previous similar message [ 5869.038175] LustreError: 249634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5874.770828] Lustre: server umount lustre-OST0000 complete [ 5876.192855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5876.195278] LustreError: Skipped 1 previous similar message [ 5879.267647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5879.271505] Lustre: Skipped 1 previous similar message [ 5882.268704] Lustre: server umount lustre-MDT0000 complete [ 5883.547254] LustreError: 247676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574120 with bad export cookie 18284478777968953267 [ 5883.552532] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5889.722345] Lustre: server umount lustre-MDT0001 complete [ 5893.305681] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5894.299441] Key type lgssc unregistered [ 5894.418399] LNet: 252798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5894.421641] LNetError: 252798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5894.433363] LNet: Removed LNI 192.168.202.109@tcp [ 5894.739129] Key type .llcrypt unregistered [ 5894.740473] Key type ._llcrypt unregistered [ 5908.682258] Key type ._llcrypt registered [ 5908.683310] Key type .llcrypt registered [ 5908.721507] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5909.174471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.190058] alg: No test for adler32 (adler32-zlib) [ 5910.043933] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 5910.131091] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5911.712168] Key type lgssc registered [ 5912.069882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5914.613360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5914.619916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.702703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5915.708708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5915.773645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.085106] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5919.910042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.943816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5919.957087] Lustre: 254977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5920.063240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5921.405727] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5923.847253] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5925.605178] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5927.824250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.904837] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5927.977769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5929.878177] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5931.349471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5932.278336] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5972.450940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5972.451770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.452661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5972.453496] LustreError: Skipped 1 previous similar message [ 5972.457052] Lustre: Skipped 1 previous similar message [ 5977.441353] Lustre: server umount lustre-OST0000 complete [ 5978.917595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5983.046603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5983.162429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5985.037709] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5987.436285] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5992.357568] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5992.362907] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5992.362927] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6023.137711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6023.138408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.140477] LustreError: Skipped 1 previous similar message [ 6023.147461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.151252] Lustre: Skipped 2 previous similar messages [ 6026.974128] Lustre: server umount lustre-OST0000 complete [ 6028.263929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.266504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6028.270381] Lustre: Skipped 2 previous similar messages [ 6033.377767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6033.379714] Lustre: Skipped 2 previous similar messages [ 6034.390449] Lustre: server umount lustre-MDT0000 complete [ 6035.597238] LustreError: 257569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574272 with bad export cookie 10101778263927636603 [ 6035.599694] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6035.602484] LustreError: 257569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6035.714528] Lustre: server umount lustre-MDT0001 complete [ 6039.591957] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6040.589570] Key type lgssc unregistered [ 6040.702345] LNet: 260388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6040.704797] LNetError: 260388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6040.716291] LNet: Removed LNI 192.168.202.109@tcp [ 6041.038701] Key type .llcrypt unregistered [ 6041.039759] Key type ._llcrypt unregistered [ 6050.276678] Key type ._llcrypt registered [ 6050.278324] Key type .llcrypt registered [ 6050.318848] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6050.681306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6050.719934] alg: No test for adler32 (adler32-zlib) [ 6051.579281] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6051.672960] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6053.264169] Key type lgssc registered [ 6053.607662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6056.231887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6056.238533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.394145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.733613] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6061.541539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.660065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6062.972241] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6065.302994] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6067.110774] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6069.358184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6069.468714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6071.395792] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6073.840402] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6083.818477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6092.257244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.259636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.263753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.306628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6094.307557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6094.308041] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6094.317880] Lustre: Skipped 1 previous similar message [ 6097.554715] Lustre: server umount lustre-OST0000 complete [ 6099.426602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.427755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6099.430061] Lustre: Skipped 1 previous similar message [ 6104.545235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.547653] Lustre: Skipped 2 previous similar messages [ 6109.665170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6109.667235] Lustre: Skipped 1 previous similar message [ 6113.248122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6113.314430] Lustre: server umount lustre-MDT0000 complete [ 6114.622488] LustreError: 261758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574351 with bad export cookie 899574948753767424 [ 6114.626569] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6114.626640] LustreError: 261758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6114.759859] Lustre: server umount lustre-MDT0001 complete [ 6118.501876] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6119.609712] Key type lgssc unregistered [ 6119.750402] LNet: 266098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6119.753251] LNetError: 266098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6119.762315] LNet: Removed LNI 192.168.202.109@tcp [ 6120.093666] Key type .llcrypt unregistered [ 6120.095381] Key type ._llcrypt unregistered [ 6127.072660] Key type ._llcrypt registered [ 6127.073703] Key type .llcrypt registered [ 6127.115385] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6127.464580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6127.496315] alg: No test for adler32 (adler32-zlib) [ 6128.350447] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6128.444070] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6130.024157] Key type lgssc registered [ 6130.415815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6133.003247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6133.008755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.159143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6135.537985] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6138.340840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6138.472556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6139.812673] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6142.275419] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6144.125110] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6146.353992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6146.473662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6148.460704] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6150.915423] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6160.807382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6165.986301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6165.987502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.988236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6165.997227] Lustre: Skipped 1 previous similar message [ 6167.264655] Lustre: server umount lustre-OST0000 complete [ 6169.056812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6169.059118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6169.063864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6169.065778] Lustre: Skipped 1 previous similar message [ 6170.652201] Lustre: server umount lustre-MDT0000 complete [ 6171.105530] LustreError: 267310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6171.106188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.110091] LustreError: 267310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6171.934139] LustreError: 269136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574408 with bad export cookie 15562173556043670241 [ 6171.936972] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6171.937382] LustreError: 269136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6172.047922] Lustre: server umount lustre-MDT0001 complete [ 6175.890972] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6177.008587] Key type lgssc unregistered [ 6177.139332] LNet: 270987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6177.141711] LNetError: 270987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6177.149279] LNet: Removed LNI 192.168.202.109@tcp [ 6177.481455] Key type .llcrypt unregistered [ 6177.482603] Key type ._llcrypt unregistered [ 6186.431170] Key type ._llcrypt registered [ 6186.432368] Key type .llcrypt registered [ 6186.470091] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6186.819310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6186.825966] alg: No test for adler32 (adler32-zlib) [ 6187.676235] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6187.763913] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6189.344114] Key type lgssc registered [ 6189.692234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6192.244772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6192.253130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.401174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6194.746069] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6197.605614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.738820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6199.095530] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6201.391354] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.222023] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6205.507323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.622645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6207.511988] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6210.028773] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6218.915781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6228.449544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6228.452286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6228.455977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6229.475576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6229.476090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6229.479984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.482396] Lustre: Skipped 1 previous similar message [ 6233.869709] Lustre: server umount lustre-OST0000 complete [ 6238.689187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6238.692517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6238.696559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6241.253718] Lustre: server umount lustre-MDT0000 complete [ 6242.522151] LustreError: 272374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574479 with bad export cookie 97724604727859204 [ 6242.525519] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6242.525812] LustreError: 272374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6242.636198] Lustre: server umount lustre-MDT0001 complete [ 6246.172597] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6247.230592] Key type lgssc unregistered [ 6247.365362] LNet: 276713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6247.367855] LNetError: 276713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6247.374319] LNet: Removed LNI 192.168.202.109@tcp [ 6247.688924] Key type .llcrypt unregistered [ 6247.689993] Key type ._llcrypt unregistered [ 6254.676116] Key type ._llcrypt registered [ 6254.677131] Key type .llcrypt registered [ 6254.712876] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6255.022214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6255.028676] alg: No test for adler32 (adler32-zlib) [ 6255.875402] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6255.965613] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6257.544184] Key type lgssc registered [ 6257.929824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6260.489633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6260.497830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6261.651209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.992644] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6265.780215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.917731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6267.241464] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6269.566401] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6271.452910] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6273.698609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.822176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6275.779100] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6278.116177] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6287.150301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6292.451488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6292.451566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.457679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6292.460493] Lustre: Skipped 1 previous similar message [ 6294.163792] Lustre: server umount lustre-OST0000 complete [ 6296.545155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6296.547472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6296.551436] Lustre: Skipped 1 previous similar message [ 6296.553417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.496502] Lustre: server umount lustre-MDT0000 complete [ 6297.570101] LustreError: 277935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.570539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.576873] LustreError: 277935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6298.750185] LustreError: 280598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574535 with bad export cookie 7652203258088609617 [ 6298.753344] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.753685] LustreError: 280598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6298.885448] Lustre: server umount lustre-MDT0001 complete [ 6302.556533] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6303.576830] Key type lgssc unregistered [ 6303.703326] LNet: 281637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6303.705592] LNetError: 281637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6303.715311] LNet: Removed LNI 192.168.202.109@tcp [ 6304.015185] Key type .llcrypt unregistered [ 6304.016185] Key type ._llcrypt unregistered [ 6307.490483] Key type ._llcrypt registered [ 6307.491658] Key type .llcrypt registered [ 6307.527780] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6312.975959] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 6318.838404] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6319.168460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6319.219104] alg: No test for adler32 (adler32-zlib) [ 6320.073212] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6320.162829] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6321.744153] Key type lgssc registered [ 6322.171670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6333.344260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6333.350158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.452607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6334.463325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6334.508408] Lustre: lustre-MDT0000: new disk, initializing [ 6334.533145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.540353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6335.975848] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6340.221851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.263333] Lustre: 285154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6340.288491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6340.291288] Lustre: Skipped 1 previous similar message [ 6340.332559] Lustre: lustre-MDT0001: new disk, initializing [ 6340.371404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6340.384655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6340.387920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6341.665237] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6345.396108] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6347.266664] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6349.425282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6349.525604] Lustre: lustre-OST0000: new disk, initializing [ 6349.527479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6349.530053] Lustre: 287123:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6349.558106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.443933] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6355.090607] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6355.451434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6355.455093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6355.491708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6356.507379] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6356.583728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6357.986370] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6358.066170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6360.545865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6360.547942] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.549377] LustreError: Skipped 1 previous similar message [ 6360.554920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.559621] Lustre: Skipped 1 previous similar message [ 6364.830424] Lustre: server umount lustre-OST0000 complete [ 6366.176996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6366.181425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.186680] Lustre: Skipped 1 previous similar message [ 6366.188373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.785532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6370.786691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.248888] Lustre: server umount lustre-MDT0000 complete [ 6373.437258] LustreError: 287122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574610 with bad export cookie 17988259714922019931 [ 6373.439966] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.440463] LustreError: 287122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6373.546768] Lustre: server umount lustre-MDT0001 complete [ 6376.163094] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:16:53 (1788574613) [ 6378.339663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6378.508171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6379.848863] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6382.635265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.758169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6384.128615] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6386.520560] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6388.377685] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6390.726108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6390.863938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6392.903951] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6395.429980] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6408.674599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6408.676855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.682847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.685667] Lustre: Skipped 1 previous similar message [ 6414.672646] Lustre: server umount lustre-OST0000 complete [ 6418.913090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6418.914232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.915333] LustreError: Skipped 2 previous similar messages [ 6418.918761] Lustre: Skipped 1 previous similar message [ 6418.919307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6418.922803] Lustre: Skipped 2 previous similar messages [ 6422.163972] Lustre: server umount lustre-MDT0000 complete [ 6423.378568] LustreError: 289512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574660 with bad export cookie 17988259714922020813 [ 6423.381801] LustreError: 290268:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6423.384693] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6423.384906] LustreError: 289512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6423.509181] Lustre: server umount lustre-MDT0001 complete [ 6427.241789] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6428.261137] Key type lgssc unregistered [ 6428.399299] LNet: 293079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6428.401928] LNetError: 293079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6428.413278] LNet: Removed LNI 192.168.202.109@tcp [ 6428.739958] Key type .llcrypt unregistered [ 6428.741862] Key type ._llcrypt unregistered [ 6437.698685] Key type ._llcrypt registered [ 6437.700536] Key type .llcrypt registered [ 6437.746649] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 6443.560605] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6443.917609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6443.933407] alg: No test for adler32 (adler32-zlib) [ 6444.768136] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6444.843068] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6446.416184] Key type lgssc registered [ 6446.715212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6461.069325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6461.075408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6462.154305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6462.163612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6462.199674] Lustre: lustre-MDT0000: new disk, initializing [ 6462.219139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6462.225315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6463.555399] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6467.677536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.717326] Lustre: 296870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6467.739165] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6467.741109] Lustre: Skipped 1 previous similar message [ 6467.777800] Lustre: lustre-MDT0001: new disk, initializing [ 6467.807575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6467.818153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6467.821185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6469.117052] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6472.723700] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6474.537111] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6476.680092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.780393] Lustre: lustre-OST2710: new disk, initializing [ 6476.782183] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6476.784741] Lustre: 298840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6476.811932] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6477.817460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6477.821700] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6477.853156] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6478.657266] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6482.247717] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6484.475198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6484.538418] Lustre: lustre-OST03e8: new disk, initializing [ 6484.541263] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6484.545453] Lustre: 299904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6484.582175] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6485.508866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6485.512106] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6485.541202] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6486.515554] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6488.917659] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.568228] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6491.632350] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6492.969747] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6493.060102] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6494.425357] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6494.502841] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6495.886117] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6495.960536] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6498.784815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6498.787707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.792644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.834175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6500.835628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.840155] Lustre: Skipped 2 previous similar messages [ 6503.135485] Lustre: server umount lustre-MDT0000 complete [ 6504.427221] LustreError: 296865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574741 with bad export cookie 6088312012393238295 [ 6504.430746] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6504.432822] LustreError: 296865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6504.548160] Lustre: server umount lustre-MDT0001 complete [ 6515.144765] Lustre: server umount lustre-OST2710 complete [ 6526.689179] Lustre: server umount lustre-OST03e8 complete [ 6532.831392] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 6535.926964] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6547.626754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6547.711841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6547.720676] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6547.757106] Lustre: lustre-MDT0000: new disk, initializing [ 6547.778686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.784620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6549.102620] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6553.136632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.174441] Lustre: 305903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6553.268264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6554.483456] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6558.110253] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6559.943510] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6562.256016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.361065] Lustre: lustre-OST0000: new disk, initializing [ 6562.362556] Lustre: Skipped 1 previous similar message [ 6562.364269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6562.366357] Lustre: Skipped 2 previous similar messages [ 6562.369134] Lustre: 307111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6562.398696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6562.400973] Lustre: Skipped 1 previous similar message [ 6564.378900] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6568.265492] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.718049] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6570.487345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6570.490655] Lustre: Skipped 1 previous similar message [ 6570.493295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6570.524231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6570.812426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6572.196301] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6572.268740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6575.590137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6575.590242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.597232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6575.600088] Lustre: Skipped 4 previous similar messages [ 6579.029047] Lustre: server umount lustre-OST0000 complete [ 6580.705156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6580.705812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6580.707986] Lustre: Skipped 2 previous similar messages [ 6580.712438] Lustre: Skipped 1 previous similar message [ 6585.826119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6585.828777] Lustre: Skipped 1 previous similar message [ 6586.459676] Lustre: server umount lustre-MDT0000 complete [ 6587.774520] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574824 with bad export cookie 6088312012393239961 [ 6587.777644] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6587.778074] LustreError: 305137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6587.907676] Lustre: server umount lustre-MDT0001 complete [ 6590.701782] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:20:28 (1788574828) [ 6591.176088] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6591.699034] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:20:29 (1788574829) [ 6592.640225] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6601.315195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.392338] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6601.399350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6601.463388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6602.903887] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6605.779616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.811268] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6605.823565] Lustre: 310970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.182302] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6622.948154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6623.023334] Lustre: Denying initial registration attempt from nid 192.168.202.109@tcp, specified as failover [ 6623.027160] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6623.029971] LustreError: 312423:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 6623.032278] LustreError: 312423:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6623.034159] LustreError: 312423:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6623.057310] Lustre: server umount lustre-OST0000 complete [ 6623.058901] LustreError: 312423:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6626.273247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6626.275347] LustreError: Skipped 1 previous similar message [ 6626.276569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6626.280891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.388838] Lustre: server umount lustre-MDT0000 complete [ 6632.661123] LustreError: 310961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574869 with bad export cookie 6088312012393240843 [ 6632.664778] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6632.664831] LustreError: 310961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6632.778329] Lustre: server umount lustre-MDT0001 complete [ 6636.398111] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6637.327945] Key type lgssc unregistered [ 6637.445291] LNet: 313539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6637.448109] LNetError: 313539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.461254] LNet: Removed LNI 192.168.202.109@tcp [ 6637.760861] Key type .llcrypt unregistered [ 6637.761832] Key type ._llcrypt unregistered [ 6640.979877] Key type ._llcrypt registered [ 6640.980946] Key type .llcrypt registered [ 6641.016631] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6641.423691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6641.432313] alg: No test for adler32 (adler32-zlib) [ 6642.266281] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6642.348806] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6643.928162] Key type lgssc registered [ 6644.320323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6668.311436] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6671.447819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.453964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6672.543437] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6672.549890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6672.606233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6674.085408] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6677.023408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.064512] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6677.079830] Lustre: 316275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6677.188426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6678.561543] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6681.087272] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6682.932388] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6698.339735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6698.408061] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6698.469815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6700.439370] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6702.849368] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6703.554283] Lustre: server umount lustre-OST0000 complete [ 6705.144284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6705.145315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6705.148255] Lustre: Skipped 1 previous similar message [ 6710.241107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.242830] Lustre: Skipped 2 previous similar messages [ 6710.804248] Lustre: server umount lustre-MDT0000 complete [ 6711.980035] LustreError: 316266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788574949 with bad export cookie 8653334257058572444 [ 6711.982912] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6711.983076] LustreError: 316266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6712.086820] Lustre: server umount lustre-MDT0001 complete [ 6715.188689] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6716.139954] Key type lgssc unregistered [ 6716.260443] LNet: 319836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6716.262738] LNetError: 319836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6716.270298] LNet: Removed LNI 192.168.202.109@tcp [ 6716.546639] Key type .llcrypt unregistered [ 6716.547674] Key type ._llcrypt unregistered [ 6725.191161] Key type ._llcrypt registered [ 6725.192219] Key type .llcrypt registered [ 6725.240466] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6725.709731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6725.721049] alg: No test for adler32 (adler32-zlib) [ 6726.564516] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6726.652942] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6728.232209] Key type lgssc registered [ 6728.571832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6730.956195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6730.962836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.095089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.347029] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6735.994027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.119762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6737.405309] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6739.652431] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6741.511833] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6745.762866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6745.875061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.823262] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6750.331582] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.224313] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 6767.073141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6767.077667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.084402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6767.586644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.590750] Lustre: Skipped 1 previous similar message [ 6769.305574] Lustre: server umount lustre-MDT0000 complete [ 6770.572405] LustreError: 321206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575007 with bad export cookie 4896118015485339719 [ 6770.575028] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6770.576185] LustreError: 321206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6771.336777] LustreError: 321218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6771.342347] LustreError: 321218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6771.343673] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 6771.347147] Lustre: Skipped 2 previous similar messages [ 6772.705227] LustreError: 321220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6772.705553] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.712763] LustreError: 321956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.712970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6775.719861] Lustre: Skipped 1 previous similar message [ 6776.747973] Lustre: server umount lustre-MDT0001 complete [ 6779.062704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6782.039199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6795.232655] LustreError: 320534:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a3f9d654 x1875456780427648/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6795.336208] LustreError: 323054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.341761] LustreError: 323054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6795.358665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.387572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6796.609374] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6799.408179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.523389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6800.548865] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6800.758576] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6802.896702] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6805.668567] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6808.422767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 6810.000973] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6810.474399] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6815.201753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6815.204070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6815.207845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6815.210472] Lustre: Skipped 1 previous similar message [ 6817.688281] Lustre: server umount lustre-OST0000 complete [ 6820.320886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6820.323828] LustreError: Skipped 1 previous similar message [ 6820.325284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.329265] Lustre: Skipped 1 previous similar message [ 6825.047455] Lustre: server umount lustre-MDT0000 complete [ 6826.252128] LustreError: 326559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575063 with bad export cookie 4896118015485346873 [ 6826.255293] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.255596] LustreError: 326559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6826.367469] Lustre: server umount lustre-MDT0001 complete [ 6830.009298] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6831.170383] Key type lgssc unregistered [ 6831.298280] LNet: 327511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6831.301180] LNetError: 327511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6831.311304] LNet: Removed LNI 192.168.202.109@tcp [ 6831.635705] Key type .llcrypt unregistered [ 6831.637058] Key type ._llcrypt unregistered [ 6840.878216] Key type ._llcrypt registered [ 6840.879393] Key type .llcrypt registered [ 6840.914227] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6841.188511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6841.195586] alg: No test for adler32 (adler32-zlib) [ 6842.072411] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6842.162453] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6843.744151] Key type lgssc registered [ 6844.178450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6846.631796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6846.637458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6847.768673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6849.205443] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6851.963848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.083784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6853.378089] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6855.740931] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6857.495755] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6862.305832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.305881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.306508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6862.314303] Lustre: Skipped 1 previous similar message [ 6863.329129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6863.331254] Lustre: Skipped 2 previous similar messages [ 6864.605322] Lustre: server umount lustre-MDT0000 complete [ 6865.754257] LustreError: 328879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575102 with bad export cookie 12871300903098312312 [ 6865.757440] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6865.759627] LustreError: 328879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6865.868873] Lustre: server umount lustre-MDT0001 complete [ 6870.352161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6870.440115] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6870.450504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6870.521064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6871.783054] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6874.518656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6874.551392] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6874.562027] Lustre: 331484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6874.651994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6875.971368] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6878.276387] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6879.955765] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6884.387137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6884.471054] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6884.474954] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6884.477866] LustreError: 333370:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 6884.480586] LustreError: 333370:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6884.483527] LustreError: 333370:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6884.508185] Lustre: server umount lustre-OST0000 complete [ 6884.509650] LustreError: 333370:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6889.315718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6889.390789] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6889.455252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6891.321837] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6893.644273] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6896.084738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.115855] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6896.137205] Lustre: lustre-OST0001: new disk, initializing [ 6896.138785] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6896.140963] Lustre: 334899:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6896.677318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6896.683023] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6896.700837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6896.732295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6898.064405] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6900.335167] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.733269] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6901.736068] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.741063] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6901.743021] Lustre: Skipped 1 previous similar message [ 6903.288931] Lustre: server umount lustre-OST0001 complete [ 6906.851276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6906.852369] LustreError: 335008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.853681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6906.853689] Lustre: Skipped 1 previous similar message [ 6906.854293] LustreError: Skipped 1 previous similar message [ 6906.855276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6906.859228] LustreError: 335008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6906.862851] Lustre: Skipped 1 previous similar message [ 6910.545202] Lustre: server umount lustre-OST0000 complete [ 6911.969590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.970330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6911.973434] Lustre: Skipped 2 previous similar messages [ 6911.976362] Lustre: Skipped 1 previous similar message [ 6922.208985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6922.211441] Lustre: Skipped 3 previous similar messages [ 6926.304124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6926.367102] Lustre: server umount lustre-MDT0000 complete [ 6927.329074] LustreError: 331498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.335503] LustreError: 331498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6927.638044] LustreError: 331479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575164 with bad export cookie 12871300903098312851 [ 6927.641498] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.642063] LustreError: 331479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6927.749426] Lustre: server umount lustre-MDT0001 complete [ 6931.679731] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6932.758790] Key type lgssc unregistered [ 6932.900367] LNet: 336858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6932.902931] LNetError: 336858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6932.909309] LNet: Removed LNI 192.168.202.109@tcp [ 6933.240212] Key type .llcrypt unregistered [ 6933.241506] Key type ._llcrypt unregistered [ 6962.425489] Key type ._llcrypt registered [ 6962.426368] Key type .llcrypt registered [ 6962.458686] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 6968.522286] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6968.913127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6968.941509] alg: No test for adler32 (adler32-zlib) [ 6969.797205] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 6969.883957] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6971.464110] Key type lgssc registered [ 6971.836912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6983.058239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6983.064634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.150447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6984.163222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6984.203130] Lustre: lustre-MDT0000: new disk, initializing [ 6984.225294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6984.231543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6985.623957] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6989.950487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6989.994705] Lustre: 341755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6990.022300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6990.024644] Lustre: Skipped 1 previous similar message [ 6990.067343] Lustre: lustre-MDT0001: new disk, initializing [ 6990.123918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6990.138760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6990.143795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6991.580719] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6995.319104] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6997.115082] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6999.388492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6999.499267] Lustre: lustre-OST0000: new disk, initializing [ 6999.501261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6999.503693] Lustre: 343724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6999.534880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.433978] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7003.072719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7003.077060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7003.134895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7005.232099] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7006.672946] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7006.750615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7008.199829] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7008.289281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7013.346238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7013.347907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.349653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7013.354087] Lustre: Skipped 1 previous similar message [ 7015.060015] Lustre: server umount lustre-OST0000 complete [ 7018.465838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7018.466548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7018.470814] Lustre: Skipped 1 previous similar message [ 7018.474416] Lustre: Skipped 2 previous similar messages [ 7022.427197] Lustre: server umount lustre-MDT0000 complete [ 7023.709325] LustreError: 341750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575260 with bad export cookie 971993077862778670 [ 7023.712875] LustreError: 341750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7023.713518] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7023.841235] Lustre: server umount lustre-MDT0001 complete [ 7026.707083] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:27:43 (1788575263) [ 7029.184113] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:27:46 (1788575266) [ 7032.050761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.216255] LustreError: 346473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7032.223624] LustreError: 346473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7032.240891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7033.711838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7036.477959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.591965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7037.897360] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7040.197172] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7041.926867] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.910928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7045.750157] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7047.940204] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.748210] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7062.229021] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7062.777951] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7063.367023] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7063.964336] Lustre: Failing over lustre-MDT0000 [ 7064.099554] Lustre: server umount lustre-MDT0000 complete [ 7067.616758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7067.619025] LustreError: Skipped 1 previous similar message [ 7067.620361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7067.624560] LustreError: 346474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.642188] LustreError: 346473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.647098] LustreError: 346473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7073.761349] LustreError: 348868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.766729] LustreError: 348868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7076.902175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.945188] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7077.037934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.040604] Lustre: Skipped 1 previous similar message [ 7077.074516] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7078.405584] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7079.048612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7082.472811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7082.482870] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7082.498648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7084.333022] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7084.845970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7086.701631] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7087.383402] Lustre: Failing over lustre-MDT0000 [ 7087.587210] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.587562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.591270] LustreError: 348958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.591280] LustreError: 348958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7087.591355] Lustre: Skipped 3 previous similar messages [ 7087.613863] Lustre: server umount lustre-MDT0000 complete [ 7097.825219] LustreError: 348868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7097.830124] LustreError: 348868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7100.611397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7100.656215] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7100.747440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7100.773923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7102.119669] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7104.648807] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7106.021412] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7106.023747] Lustre: Skipped 2 previous similar messages [ 7106.032424] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 7106.049622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7107.892403] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7108.422713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7111.138445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7111.139111] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7111.140101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7111.145315] Lustre: Skipped 3 previous similar messages [ 7116.257816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7116.259875] Lustre: Skipped 4 previous similar messages [ 7116.762853] Lustre: server umount lustre-MDT0000 complete [ 7118.046341] LustreError: 348311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575355 with bad export cookie 971993077862781365 [ 7118.048695] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7118.049672] LustreError: 348311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7118.162426] Lustre: server umount lustre-MDT0001 complete [ 7129.683313] Lustre: server umount lustre-OST0000 complete [ 7134.899138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.090484] LustreError: 352824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7135.098588] LustreError: 352824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7135.117746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.575619] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7139.486177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7141.085428] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7143.588694] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7147.348241] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7150.171230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7152.258397] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7154.707834] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7161.574490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7166.945451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7166.949209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.955938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7172.371857] Lustre: server umount lustre-OST0000 complete [ 7175.137952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7175.139706] Lustre: Skipped 3 previous similar messages [ 7179.666970] Lustre: server umount lustre-MDT0000 complete [ 7180.257845] LustreError: 353553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7180.262041] LustreError: 353553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7180.811934] LustreError: 352850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575417 with bad export cookie 971993077862782086 [ 7180.814664] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7180.815670] LustreError: 352850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7180.915584] Lustre: server umount lustre-MDT0001 complete [ 7184.069870] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7184.979518] Key type lgssc unregistered [ 7185.105376] LNet: 356375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7185.107776] LNetError: 356375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7185.116264] LNet: Removed LNI 192.168.202.109@tcp [ 7185.383723] Key type .llcrypt unregistered [ 7185.384763] Key type ._llcrypt unregistered [ 7194.408262] Key type ._llcrypt registered [ 7194.409189] Key type .llcrypt registered [ 7194.443652] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7194.717208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7194.742801] alg: No test for adler32 (adler32-zlib) [ 7195.587529] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7195.666556] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7197.240189] Key type lgssc registered [ 7197.607272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7200.312209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7200.318343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.478788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.975018] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7205.921423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.037947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.389519] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7209.681828] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7211.314329] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7213.383028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.481259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7215.208078] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7217.413600] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7226.793901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7237.090420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7237.091177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.094054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7237.098120] Lustre: Skipped 1 previous similar message [ 7239.826913] Lustre: server umount lustre-MDT0000 complete [ 7242.209261] LustreError: 357862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7242.213241] LustreError: 357862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7243.261090] LustreError: 357844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575480 with bad export cookie 18373848498241319734 [ 7243.264348] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7243.265884] LustreError: 357844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7243.388739] Lustre: server umount lustre-MDT0001 complete [ 7254.947429] Lustre: server umount lustre-OST0000 complete [ 7259.267186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.405880] LustreError: 361593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.427957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.731109] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7263.432013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.543878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7264.811265] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7267.039264] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7268.726210] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.828985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.701573] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7274.871975] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.261970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7294.433547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7294.434215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.435472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7294.435477] Lustre: Skipped 2 previous similar messages [ 7294.440839] Lustre: Skipped 2 previous similar messages [ 7298.198350] Lustre: server umount lustre-MDT0000 complete [ 7299.380256] LustreError: 361618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575536 with bad export cookie 18373848498241320833 [ 7299.382537] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.388649] LustreError: 361618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7299.504245] Lustre: server umount lustre-MDT0001 complete [ 7310.935599] Lustre: server umount lustre-OST0000 complete [ 7315.751129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.900691] LustreError: 365520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.922632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.925217] Lustre: Skipped 1 previous similar message [ 7317.208174] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7319.870416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.236353] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7323.485455] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7325.233601] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7327.339319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7329.264836] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7331.685562] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7340.773879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7346.147159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7346.147352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7346.148025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.148030] Lustre: Skipped 2 previous similar messages [ 7346.150046] LustreError: Skipped 1 previous similar message [ 7346.158150] Lustre: Skipped 2 previous similar messages [ 7349.526658] Lustre: server umount lustre-OST0000 complete [ 7350.752691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7350.754948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.758389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7350.760343] Lustre: Skipped 1 previous similar message [ 7356.385051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7356.387136] Lustre: Skipped 3 previous similar messages [ 7356.824949] Lustre: server umount lustre-MDT0000 complete [ 7358.049327] LustreError: 366255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575595 with bad export cookie 18373848498241321841 [ 7358.052401] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.054483] LustreError: 366255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7358.162184] Lustre: server umount lustre-MDT0001 complete [ 7361.744530] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7362.801197] Key type lgssc unregistered [ 7362.934337] LNet: 369069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.937299] LNetError: 369069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.946299] LNet: Removed LNI 192.168.202.109@tcp [ 7363.229654] Key type .llcrypt unregistered [ 7363.231102] Key type ._llcrypt unregistered [ 7372.821851] Key type ._llcrypt registered [ 7372.822973] Key type .llcrypt registered [ 7372.858456] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7373.257825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7373.266889] alg: No test for adler32 (adler32-zlib) [ 7374.126546] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7374.214702] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7375.792185] Key type lgssc registered [ 7376.177564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7378.926758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7378.933740] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.937660] LustreError: 370481:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7378.947115] LustreError: 370481:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7378.950209] LustreError: 370481:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7378.953663] LustreError: 370481:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7378.957541] LustreError: 370481:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7381.647761] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7381.651698] LustreError: 370878:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7381.661336] LustreError: 370878:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7381.665060] LustreError: 370878:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7381.668604] LustreError: 370878:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7381.671623] LustreError: 370878:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7388.211570] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7389.332364] Key type lgssc unregistered [ 7389.461423] LNet: 371879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7389.464663] LNetError: 371879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7389.471305] LNet: Removed LNI 192.168.202.109@tcp [ 7389.797317] Key type .llcrypt unregistered [ 7389.799185] Key type ._llcrypt unregistered [ 7396.423571] Key type ._llcrypt registered [ 7396.424669] Key type .llcrypt registered [ 7396.461530] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 7403.183206] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7403.499659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7403.510128] alg: No test for adler32 (adler32-zlib) [ 7404.381552] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7404.474861] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7406.056193] Key type lgssc registered [ 7406.395258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7417.479320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7417.484636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7418.570337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7418.583385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7418.620679] Lustre: lustre-MDT0000: new disk, initializing [ 7418.651755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.658788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7420.078361] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7424.507928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.549454] Lustre: 375003:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7424.570904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7424.573070] Lustre: Skipped 1 previous similar message [ 7424.606943] Lustre: lustre-MDT0001: new disk, initializing [ 7424.641280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7424.654123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7424.657714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7426.049345] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7430.061470] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7432.190551] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7434.662890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7434.769253] Lustre: lustre-OST0000: new disk, initializing [ 7434.771228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7434.773969] Lustre: 376971:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7434.809584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.884798] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7440.846436] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.402577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7441.406622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7441.447275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7442.388357] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7442.476371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7444.054428] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7444.148591] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7446.497143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7446.500386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7446.501609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7446.507259] Lustre: Skipped 1 previous similar message [ 7446.510735] Lustre: Skipped 1 previous similar message [ 7451.023492] Lustre: server umount lustre-OST0000 complete [ 7455.713091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7455.715482] LustreError: Skipped 1 previous similar message [ 7455.716585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7455.720473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7456.737821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7456.738160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.489325] Lustre: server umount lustre-MDT0000 complete [ 7460.926864] LustreError: 374996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575697 with bad export cookie 1393413476626960298 [ 7460.929091] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7460.932186] LustreError: 374996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7461.055987] Lustre: server umount lustre-MDT0001 complete [ 7464.036361] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7464.605106] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:35:01 (1788575701) [ 7466.992865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7467.221995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.871763] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7472.609215] LustreError: 379426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7472.616715] LustreError: 379426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7472.815792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.979796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7475.873414] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7478.885573] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7481.218208] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7483.863844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7483.984278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.836838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7488.245412] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7490.722670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7490.784898] Lustre: lustre-OST0001: new disk, initializing [ 7490.786731] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7490.789738] Lustre: 382259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7492.675473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7492.680211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7492.720588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7492.927581] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7497.029435] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7501.185125] Lustre: server umount lustre-OST0001 complete [ 7501.874471] LustreError: 381248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.817623] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7502.822084] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7503.332419] LustreError: 381246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.339598] LustreError: 381246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7508.453814] LustreError: 382566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7508.458625] LustreError: 382566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7513.569946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7513.570923] LustreError: 382566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7513.571152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7513.571159] LustreError: Skipped 1 previous similar message [ 7513.571572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7513.571577] Lustre: Skipped 1 previous similar message [ 7513.575174] Lustre: Skipped 2 previous similar messages [ 7513.578993] LustreError: 382566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7518.690127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.692880] Lustre: Skipped 2 previous similar messages [ 7519.518900] Lustre: server umount lustre-OST0000 complete [ 7523.810426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.815880] Lustre: Skipped 1 previous similar message [ 7528.929375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.932138] Lustre: Skipped 3 previous similar messages [ 7535.072252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7535.134530] Lustre: server umount lustre-MDT0000 complete [ 7536.355050] LustreError: 381268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575773 with bad export cookie 1393413476626961166 [ 7536.357930] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7536.359641] LustreError: 381268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7542.513439] Lustre: server umount lustre-MDT0001 complete [ 7546.266761] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7547.285268] Key type lgssc unregistered [ 7547.421319] LNet: 384244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.423618] LNetError: 384244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.430273] LNet: Removed LNI 192.168.202.109@tcp [ 7547.754361] Key type .llcrypt unregistered [ 7547.755856] Key type ._llcrypt unregistered [ 7564.812778] Key type ._llcrypt registered [ 7564.814307] Key type .llcrypt registered [ 7564.852568] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7565.299913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7565.330276] alg: No test for adler32 (adler32-zlib) [ 7566.174411] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7566.258588] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7567.840182] Key type lgssc registered [ 7568.211433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7570.799076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7570.805929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7571.904915] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7571.912301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7571.975520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.484526] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7576.468405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.509090] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7576.520352] Lustre: 387013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7576.618355] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.059179] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7580.529985] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7582.312298] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7585.249343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.250204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7585.254542] Lustre: Skipped 1 previous similar message [ 7589.142643] Lustre: server umount lustre-MDT0000 complete [ 7590.511445] LustreError: 387005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575827 with bad export cookie 17621376808166342092 [ 7590.515030] LustreError: 387005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7590.515277] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7590.637768] Lustre: server umount lustre-MDT0001 complete [ 7593.987528] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:37:11 (1788575831) [ 7596.181581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.351975] LustreError: 389521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.357021] LustreError: 389521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7596.377394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7597.847747] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7600.956328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.094649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.654932] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7605.520362] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7607.882902] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7610.570896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7610.654191] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7610.733731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.046305] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7615.613172] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7619.652334] Lustre: Permanently deactivating lustre-OST0000 [ 7619.658591] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7626.233455] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7626.987293] LustreError: 392555:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7626.992993] LustreError: 392555:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7626.996152] LustreError: 392555:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7627.735304] LustreError: 392603:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7627.738415] LustreError: 392603:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7634.594688] Lustre: server umount lustre-OST0000 complete [ 7635.600036] LustreError: 392852:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 7635.603073] LustreError: 392852:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7636.964086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7636.973378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7636.985443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7636.988413] Lustre: Skipped 1 previous similar message [ 7642.593026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7642.595017] Lustre: Skipped 3 previous similar messages [ 7646.715865] Lustre: server umount lustre-MDT0000 complete [ 7647.713159] LustreError: 392019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.719954] LustreError: 392019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7648.094277] LustreError: 392211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575885 with bad export cookie 17621376808166342659 [ 7648.097441] LustreError: 392211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7648.098703] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7658.528965] Lustre: server umount lustre-MDT0001 complete [ 7662.221457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.041962] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7665.420969] LustreError: 394009:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7668.592355] Lustre: 394347:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.109@tcp [ 7669.782920] Lustre: server umount MGS complete [ 7675.763609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7675.993308] LustreError: 395071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.002815] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7676.005181] Lustre: Skipped 1 previous similar message [ 7676.032225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.828188] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7681.231302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.343392] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7682.865531] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7685.396422] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7687.347365] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7688.102679] Lustre: Permanently reactivating lustre-OST0000 [ 7688.108131] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7688.111124] Lustre: Skipped 2 previous similar messages [ 7710.177468] LustreError: 396753:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7713.352464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.478807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.481061] Lustre: Skipped 1 previous similar message [ 7715.612266] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7718.602076] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7731.170460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7731.171335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.174028] LustreError: Skipped 1 previous similar message [ 7731.177903] Lustre: Skipped 1 previous similar message [ 7731.178644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7731.183461] Lustre: Skipped 1 previous similar message [ 7732.951544] Lustre: server umount lustre-OST0000 complete [ 7736.289573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.293157] Lustre: Skipped 1 previous similar message [ 7736.295268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.297635] Lustre: Skipped 1 previous similar message [ 7736.663063] Lustre: server umount lustre-MDT0000 complete [ 7738.014179] LustreError: 396644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788575975 with bad export cookie 17621376808166343695 [ 7738.017809] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7738.019244] LustreError: 396644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7738.129380] Lustre: server umount lustre-MDT0001 complete [ 7741.928739] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7743.034711] Key type lgssc unregistered [ 7743.167420] LNet: 399596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7743.170077] LNetError: 399596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7743.176322] LNet: Removed LNI 192.168.202.109@tcp [ 7743.509421] Key type .llcrypt unregistered [ 7743.510582] Key type ._llcrypt unregistered [ 7749.639466] Key type ._llcrypt registered [ 7749.640914] Key type .llcrypt registered [ 7749.684307] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 7756.020582] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7756.431202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7756.442274] alg: No test for adler32 (adler32-zlib) [ 7757.288315] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7757.369177] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7758.952113] Key type lgssc registered [ 7759.267301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7771.695097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7771.702530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7772.792383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7772.804227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7772.844434] Lustre: lustre-MDT0000: new disk, initializing [ 7772.871815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.880833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7774.710684] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7780.864085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.910534] Lustre: 402720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7780.935972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7780.938419] Lustre: Skipped 1 previous similar message [ 7780.976935] Lustre: lustre-MDT0001: new disk, initializing [ 7781.020951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.039631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7781.046821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7782.827286] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7787.567163] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7790.167295] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.256848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7793.380192] Lustre: lustre-OST0000: new disk, initializing [ 7793.382118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7793.385293] Lustre: 404688:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7793.418055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7796.009302] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7796.104371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7796.108240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7796.175433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7800.213385] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.855457] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7801.944943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7803.612814] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7803.696327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7806.434025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7806.435094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.437265] LustreError: Skipped 1 previous similar message [ 7806.442498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7806.445955] Lustre: Skipped 1 previous similar message [ 7810.532315] Lustre: server umount lustre-OST0000 complete [ 7812.065349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7812.068532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.073381] Lustre: Skipped 1 previous similar message [ 7812.076341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.673191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.673488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.793221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7821.795852] Lustre: Skipped 2 previous similar messages [ 7826.400232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7826.489914] Lustre: server umount lustre-MDT0000 complete [ 7828.230391] LustreError: 402714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576065 with bad export cookie 15077722773117907767 [ 7828.232309] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.241324] LustreError: 402714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7828.395891] Lustre: server umount lustre-MDT0001 complete [ 7832.258139] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7832.988950] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:41:10 (1788576070) [ 7836.365878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7836.581220] LustreError: 407145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.591469] LustreError: 407145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7836.612068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.953918] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7841.761421] LustreError: 407146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7846.010602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7846.243317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7848.338494] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7851.849484] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7854.950075] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7858.746231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7858.946930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7862.561772] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7866.940643] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7867.914189] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7869.878384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7881.699530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7881.701538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.705537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.711612] Lustre: Skipped 1 previous similar message [ 7881.720082] Lustre: Skipped 1 previous similar message [ 7884.315641] Lustre: server umount lustre-OST0000 complete [ 7886.819943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.834194] Lustre: Skipped 1 previous similar message [ 7891.937625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7891.939934] Lustre: Skipped 3 previous similar messages [ 7892.517933] Lustre: server umount lustre-MDT0000 complete [ 7894.814200] LustreError: 407127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576131 with bad export cookie 15077722773117908649 [ 7894.814869] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7894.817672] LustreError: 407127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7895.006224] Lustre: server umount lustre-MDT0001 complete [ 7901.665876] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7903.230610] Key type lgssc unregistered [ 7903.393562] LNet: 410865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.399912] LNetError: 410865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.408949] LNet: Removed LNI 192.168.202.109@tcp [ 7903.984641] Key type .llcrypt unregistered [ 7903.985786] Key type ._llcrypt unregistered [ 7921.911032] Key type ._llcrypt registered [ 7921.913123] Key type .llcrypt registered [ 7921.994415] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7922.240362] Key type .llcrypt unregistered [ 7922.243361] Key type ._llcrypt unregistered [ 7934.395839] Key type ._llcrypt registered [ 7934.397759] Key type .llcrypt registered [ 7934.474606] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7935.276174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7935.293860] alg: No test for adler32 (adler32-zlib) [ 7936.314368] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 7936.526483] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7938.160159] Key type lgssc registered [ 7939.231244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7945.845367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7945.857136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.146496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7949.468714] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7955.387811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7955.623855] LustreError: 413047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.639357] LustreError: 413047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7955.651661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.678078] LustreError: 413048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7957.695601] LustreError: 413048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7957.709594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7962.816099] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7963.151786] LustreError: 413588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.171498] LustreError: 413588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7968.233572] LustreError: 413042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.260994] LustreError: 413042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7972.949809] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7973.353672] LustreError: 413047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7973.376696] LustreError: 413047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7980.628850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7980.995488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7984.990230] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8001.510706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.518788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8001.524655] Lustre: Skipped 1 previous similar message [ 8001.548725] Lustre: Skipped 1 previous similar message [ 8006.254850] Lustre: server umount lustre-OST0000 complete [ 8011.746527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8011.748126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8011.753383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8011.755979] Lustre: Skipped 1 previous similar message [ 8011.770362] Lustre: Skipped 1 previous similar message [ 8016.870094] LustreError: 413044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8016.885459] LustreError: 413044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8017.073493] Lustre: server umount lustre-MDT0000 complete [ 8021.382236] LustreError: 413028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576258 with bad export cookie 14393945065340787237 [ 8021.390207] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8021.397941] LustreError: 413028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8022.277668] Lustre: server umount lustre-MDT0001 complete [ 8038.776441] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8044.456614] Key type lgssc unregistered [ 8044.867847] LNet: 416207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.877909] LNetError: 416207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.903265] LNet: Removed LNI 192.168.202.109@tcp [ 8046.681598] Key type .llcrypt unregistered [ 8046.691338] Key type ._llcrypt unregistered [ 8085.543282] Key type ._llcrypt registered [ 8085.554726] Key type .llcrypt registered [ 8085.766548] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8087.231318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8087.266153] alg: No test for adler32 (adler32-zlib) [ 8088.377199] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8088.774063] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8090.504154] Key type lgssc registered [ 8092.094854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8101.975458] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8102.002671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8117.664314] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8122.787493] LustreError: 417595:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 8149.409180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8155.648582] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8163.092950] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8169.975647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.502397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.591685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8175.280517] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8175.597908] LustreError: 417621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.624724] LustreError: 417621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8180.707681] LustreError: 417620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8184.151323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8184.450337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8189.503204] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8197.606683] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8203.782276] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8211.436530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8211.445735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8211.463969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8213.188824] Lustre: server umount lustre-OST0000 complete [ 8219.119962] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8219.136555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.145955] Lustre: Skipped 1 previous similar message [ 8219.150740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8219.160108] Lustre: Skipped 1 previous similar message [ 8221.674590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.677162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.700028] Lustre: Skipped 1 previous similar message [ 8223.429207] Lustre: server umount lustre-MDT0000 complete [ 8226.794279] LustreError: 418635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.814164] LustreError: 418635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8227.604743] LustreError: 417601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576464 with bad export cookie 5510198130436979576 [ 8227.606197] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8227.619223] LustreError: 417601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8228.169526] Lustre: server umount lustre-MDT0001 complete [ 8240.797706] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8244.329596] Key type lgssc unregistered [ 8244.588272] LNet: 421165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8244.619502] LNetError: 421165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8245.676645] LNet: Removed LNI 192.168.202.109@tcp [ 8246.826701] Key type .llcrypt unregistered [ 8246.829410] Key type ._llcrypt unregistered [ 8273.788309] Key type ._llcrypt registered [ 8273.790103] Key type .llcrypt registered [ 8273.955691] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8275.072417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8275.232485] alg: No test for adler32 (adler32-zlib) [ 8276.238991] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8276.451899] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8278.088227] Key type lgssc registered [ 8279.454855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.913450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8287.925524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.450572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.820366] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8298.982947] LustreError: 422579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8299.000689] LustreError: 422579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8304.119257] LustreError: 422580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.703473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8305.524664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8311.894035] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8322.898625] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8331.459809] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8342.069544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.783644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8351.069187] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8359.168087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8359.926822] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8367.109265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8367.113713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.131129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.791529] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 8368.026167] Lustre: server umount lustre-MDT0000 complete [ 8369.640355] LustreError: 422580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8369.649148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.667334] LustreError: 422580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8369.733240] Lustre: Skipped 1 previous similar message [ 8370.834462] LustreError: 425159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.9@tcp arrived at 1788576607 with bad export cookie 3152167267074946515 [ 8373.043386] LustreError: 422561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788576610 with bad export cookie 3152167267074945619 [ 8373.054964] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.058556] LustreError: 422561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8374.758177] LustreError: 422579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8374.783269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.796572] LustreError: 422579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8374.859282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.792680] LustreError: 423515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8378.805099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8378.834359] LustreError: 423515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8379.448261] Lustre: server umount lustre-OST0000 complete [ 8388.365649] Lustre: server umount lustre-MDT0001 complete [ 8400.332988] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8403.649313] Key type lgssc unregistered [ 8404.021991] LNet: 426264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8404.037471] LNetError: 426264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8404.068668] LNet: Removed LNI 192.168.202.109@tcp [ 8404.986492] Key type .llcrypt unregistered [ 8404.990061] Key type ._llcrypt unregistered [ 8432.784741] Key type ._llcrypt registered [ 8432.787256] Key type .llcrypt registered [ 8432.904475] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8434.687881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8434.784063] alg: No test for adler32 (adler32-zlib) [ 8435.886338] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8436.149703] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8437.816924] Key type lgssc registered [ 8439.120183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8447.076537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8447.128209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.917503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.104717] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8458.221947] LustreError: 427678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8458.252044] LustreError: 427678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8463.332346] LustreError: 427677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8465.099417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8465.423808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8470.848914] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8478.774877] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8483.924988] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8491.079258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8491.375834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.437645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8498.732841] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8507.098894] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8515.225783] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 8516.069653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.094254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8518.662959] Lustre: server umount lustre-MDT0001 complete [ 8520.334279] LustreError: 427672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.309520] LustreError: 429434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.325604] LustreError: 429434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8529.377942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8529.392683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.406747] Lustre: Skipped 1 previous similar message [ 8529.415935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8529.428259] Lustre: Skipped 1 previous similar message [ 8531.430193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8531.430631] LustreError: 428623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.622291] Lustre: server umount lustre-OST0000 complete [ 8537.623433] Lustre: server umount lustre-MDT0000 complete [ 8551.654672] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8554.986274] Key type lgssc unregistered [ 8555.333636] LNet: 431363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8555.344480] LNetError: 431363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8555.368987] LNet: Removed LNI 192.168.202.109@tcp [ 8556.665250] Key type .llcrypt unregistered [ 8556.667163] Key type ._llcrypt unregistered [ 8584.052608] Key type ._llcrypt registered [ 8584.061863] Key type .llcrypt registered [ 8584.211944] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 8601.512722] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8603.993266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8604.058484] alg: No test for adler32 (adler32-zlib) [ 8605.467468] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8605.908940] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8607.740905] Key type lgssc registered [ 8609.939919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8657.791717] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8675.232869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8675.275924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.534987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8676.561388] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8676.638712] Lustre: lustre-MDT0000: new disk, initializing [ 8676.729076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8676.746922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8681.897684] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8700.422462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8700.652893] Lustre: lustre-OST0000: new disk, initializing [ 8700.656563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8700.660743] Lustre: Skipped 1 previous similar message [ 8700.665915] Lustre: 436412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8700.740252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8707.724738] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8708.712400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8708.737648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8708.776642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8719.161830] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8737.502255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8737.730702] Lustre: 435391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8737.829891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8737.979728] Lustre: lustre-MDT0001: new disk, initializing [ 8738.282174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8738.333899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8738.355104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8743.568909] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8754.358232] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8754.659939] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8759.578917] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8759.927168] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8766.441374] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8766.457516] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8766.479250] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8769.199430] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 8771.048770] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.059574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8771.075711] Lustre: Skipped 1 previous similar message [ 8774.307585] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 8779.413158] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 8779.427046] Lustre: Skipped 2 previous similar messages [ 8780.256510] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8780.499412] Lustre: server umount lustre-MDT0001 complete [ 8786.401750] LustreError: 435404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.423624] LustreError: 435404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8791.540520] LustreError: 435405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.569427] LustreError: 435405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8796.642766] LustreError: 435399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.653926] LustreError: 435399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8800.742964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8800.752289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8800.764325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8801.765170] LustreError: 435405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.532574] Lustre: server umount lustre-OST0000 complete [ 8806.883318] LustreError: 438111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.097466] Lustre: server umount lustre-MDT0000 complete [ 8830.235403] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8833.082108] Key type lgssc unregistered [ 8833.407506] LNet: 440166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8833.418051] LNetError: 440166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8834.474062] LNet: Removed LNI 192.168.202.109@tcp [ 8835.536983] Key type .llcrypt unregistered [ 8835.539559] Key type ._llcrypt unregistered [ 8881.036769] Key type ._llcrypt registered [ 8881.040675] Key type .llcrypt registered [ 8881.174607] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8881.620614] Key type .llcrypt unregistered [ 8881.622349] Key type ._llcrypt unregistered [ 8900.974590] Key type ._llcrypt registered [ 8900.977377] Key type .llcrypt registered [ 8901.085341] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8902.548579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8902.711745] alg: No test for adler32 (adler32-zlib) [ 8903.947673] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [ 8904.300752] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8906.100472] Key type lgssc registered [ 8907.705444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8937.749608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8937.768395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8939.126971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8939.184477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8939.323788] Lustre: lustre-MDT0000: new disk, initializing [ 8939.420513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.452849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8944.127924] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8955.313903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8955.502268] Lustre: 443552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8955.573259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8955.577228] Lustre: Skipped 1 previous similar message [ 8955.674238] Lustre: lustre-MDT0001: new disk, initializing [ 8955.863546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.934554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8955.957271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8961.132505] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8971.110716] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8977.833431] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8985.224264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8985.582338] Lustre: lustre-OST0000: new disk, initializing [ 8985.585955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8985.593614] Lustre: 445518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8985.731919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8992.892897] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 8993.916385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8993.931651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8994.053050] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9003.238429] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9012.197204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9012.209128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9012.234153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9014.754487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.758646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9014.772525] Lustre: Skipped 1 previous similar message [ 9014.786204] Lustre: Skipped 1 previous similar message [ 9017.352827] Lustre: server umount lustre-MDT0000 complete [ 9021.150204] LustreError: 443543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577258 with bad export cookie 6694887008620131279 [ 9021.157378] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.163768] LustreError: 443543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9021.468534] Lustre: server umount lustre-MDT0001 complete [ 9035.360767] Lustre: server umount lustre-OST0000 complete [ 9046.448299] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:01:22 (1788577282) [ 9054.867135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9055.520443] LustreError: 447559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9055.547605] LustreError: 447559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9055.623389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.834472] LustreError: 447560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9061.172364] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9065.956113] LustreError: 447559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9070.493348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.920650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9075.701387] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9083.226257] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9089.007512] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9109.097725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9109.380030] Lustre: Found index 0 for lustre-OST0000, updating log [ 9109.581435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.136239] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9123.459452] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9127.037513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9136.617221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9136.633040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.657963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9136.663155] Lustre: Skipped 1 previous similar message [ 9139.542984] Lustre: server umount lustre-OST0000 complete [ 9158.113525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9158.118470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.126427] Lustre: Skipped 1 previous similar message [ 9158.129841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9158.135171] Lustre: Skipped 3 previous similar messages [ 9162.258152] Lustre: server umount lustre-MDT0000 complete [ 9163.236990] LustreError: 447559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9163.263119] LustreError: 447559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9165.972970] LustreError: 447539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788577403 with bad export cookie 6694887008620132672 [ 9165.976306] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9165.984572] LustreError: 447539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9166.353700] Lustre: server umount lustre-MDT0001 complete [ 9167.525887] LNet: 1 local NIs in recovery (showing 1): 192.168.202.109@tcp [ 9175.040443] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:03:31 (1788577411) [ 9798.671741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.267017] LustreError: 453973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.280782] LustreError: 453973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9799.338291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.777751] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9804.775555] LustreError: 453974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.899832] LustreError: 453973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.207666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9812.496735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9816.918547] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9823.583662] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9829.898712] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9850.169600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9850.397193] Lustre: Found index 0 for lustre-OST0000, updating log [ 9850.828390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9857.715100] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 9858.786134] LNet: 1 local NIs in recovery (showing 1): 192.168.202.109@tcp [ 9865.328869] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9867.850301] Lustre: server umount lustre-OST0000 complete [ 9872.355627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9872.357816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9872.370234] Lustre: Skipped 2 previous similar messages [ 9872.386049] Lustre: Skipped 3 previous similar messages [ 9873.442461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9873.446478] Lustre: Skipped 1 previous similar message [ 9874.467986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9874.479509] Lustre: Skipped 1 previous similar message [ 9876.513841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9876.518832] Lustre: Skipped 3 previous similar messages [ 9877.546487] LustreError: 453968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9877.562589] LustreError: 453968:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9877.607725] Lustre: server umount lustre-MDT0000 complete [ 9881.200286] LustreError: 453953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578118 with bad export cookie 6694887008620133757 [ 9881.218106] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9881.510263] Lustre: server umount lustre-MDT0001 complete [ 9914.155504] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 9924.815409] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 9960.009546] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:16:36 (1788578196) [ 9978.575869] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 9981.829944] Key type lgssc unregistered [ 9982.084452] LNet: 460580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9982.090550] LNetError: 460580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10110.127378] LNet: Removed LNI 192.168.202.109@tcp [10111.016739] Key type .llcrypt unregistered [10111.018776] Key type ._llcrypt unregistered [10135.154134] Key type ._llcrypt registered [10135.155462] Key type .llcrypt registered [10135.283553] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [10136.899068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10137.006888] alg: No test for adler32 (adler32-zlib) [10138.183382] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [10138.334921] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [10139.936178] Key type lgssc registered [10141.523100] Lustre: Echo OBD driver; http://www.lustre.org/ [10148.169940] LNet: Added LNI 192.168.202.109@tcp100 [8/256/0/180] [10149.104651] LNet: Added LNI 192.168.202.109@tcp101 [8/256/0/180] [10150.106524] LNet: Added LNI 192.168.202.109@tcp102 [8/256/0/180] [10151.259420] LNet: Added LNI 192.168.202.109@tcp103 [8/256/0/180] [10152.240679] LNet: Added LNI 192.168.202.109@tcp104 [8/256/0/180] [10153.270153] LNet: Added LNI 192.168.202.109@tcp105 [8/256/0/180] [10154.329093] LNet: Added LNI 192.168.202.109@tcp106 [8/256/0/180] [10155.374641] LNet: Added LNI 192.168.202.109@tcp107 [8/256/0/180] [10156.348887] LNet: Added LNI 192.168.202.109@tcp108 [8/256/0/180] [10157.312305] LNet: Added LNI 192.168.202.109@tcp109 [8/256/0/180] [10158.194596] LNet: Added LNI 192.168.202.109@tcp110 [8/256/0/180] [10158.986729] LNet: Added LNI 192.168.202.109@tcp111 [8/256/0/180] [10159.930559] LNet: Added LNI 192.168.202.109@tcp112 [8/256/0/180] [10160.765068] LNet: Added LNI 192.168.202.109@tcp113 [8/256/0/180] [10161.702318] LNet: Added LNI 192.168.202.109@tcp114 [8/256/0/180] [10162.994957] LNet: Added LNI 192.168.202.109@tcp115 [8/256/0/180] [10164.024381] LNet: Added LNI 192.168.202.109@tcp116 [8/256/0/180] [10165.201990] LNet: Added LNI 192.168.202.109@tcp117 [8/256/0/180] [10166.347605] LNet: Added LNI 192.168.202.109@tcp118 [8/256/0/180] [10167.336902] LNet: Added LNI 192.168.202.109@tcp119 [8/256/0/180] [10168.316831] LNet: Added LNI 192.168.202.109@tcp120 [8/256/0/180] [10169.416570] LNet: Added LNI 192.168.202.109@tcp121 [8/256/0/180] [10170.397882] LNet: Added LNI 192.168.202.109@tcp122 [8/256/0/180] [10171.410199] LNet: Added LNI 192.168.202.109@tcp123 [8/256/0/180] [10172.260177] LNet: Added LNI 192.168.202.109@tcp124 [8/256/0/180] [10173.309277] LNet: Added LNI 192.168.202.109@tcp125 [8/256/0/180] [10174.120583] LNet: Added LNI 192.168.202.109@tcp126 [8/256/0/180] [10175.470596] LNet: Added LNI 192.168.202.109@tcp127 [8/256/0/180] [10176.490814] LNet: Added LNI 192.168.202.109@tcp128 [8/256/0/180] [10177.599972] LNet: Added LNI 192.168.202.109@tcp129 [8/256/0/180] [10178.615765] LNet: Added LNI 192.168.202.109@tcp130 [8/256/0/180] [10179.698688] LNet: Added LNI 192.168.202.109@tcp131 [8/256/0/180] [10180.781883] LNet: Added LNI 192.168.202.109@tcp132 [8/256/0/180] [10181.541567] LNet: Added LNI 192.168.202.109@tcp133 [8/256/0/180] [10182.370600] LNet: Added LNI 192.168.202.109@tcp134 [8/256/0/180] [10183.201748] LNet: Added LNI 192.168.202.109@tcp135 [8/256/0/180] [10190.071243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10190.132933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10191.309120] Lustre: 463614:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.109@tcp: can't add NID 192.168.202.109@tcp130: rc = -75 [10191.411138] LNetError: 461071:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10191.431212] LNetError: Protocol error connecting to 192.168.202.109@tcp117 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10191.447984] LNetError: Skipped 10 previous similar messages [10191.491325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10191.521787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10191.586475] Lustre: lustre-MDT0000: new disk, initializing [10191.648118] LNet: 7 local NIs in recovery (showing 7): 192.168.202.109@tcp123, 192.168.202.109@tcp124, 192.168.202.109@tcp125, 192.168.202.109@tcp126, 192.168.202.109@tcp128, 192.168.202.109@tcp127, 192.168.202.109@tcp129 [10191.667300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10191.683463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10192.480849] LNetError: 461070:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10192.486601] LNetError: 461070:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [10192.493527] LNetError: Protocol error connecting to 192.168.202.109@tcp110 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10192.506670] LNetError: Skipped 1 previous similar message [10194.786490] LNetError: 461069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10194.800524] LNetError: 461069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [10194.838475] LNetError: Protocol error connecting to 192.168.202.109@tcp123 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10194.858722] LNetError: Skipped 9 previous similar messages [10196.337270] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10198.882352] LNetError: 461068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10198.890449] LNetError: 461068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [10198.895564] LNetError: Protocol error connecting to 192.168.202.109@tcp123 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10206.053731] LNetError: 461071:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10206.066436] LNetError: Protocol error connecting to 192.168.202.109@tcp123 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10208.087255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10208.277879] Lustre: 463630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10208.398798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10208.403802] Lustre: Skipped 1 previous similar message [10208.540926] Lustre: lustre-MDT0001: new disk, initializing [10208.618139] Lustre: 464432:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.109@tcp: can't add NID 192.168.202.109@tcp130: rc = -75 [10208.625679] Lustre: 464432:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10208.823674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10208.891637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10208.903743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10213.703433] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10218.561170] Lustre: 464863:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.109@tcp: can't add NID 192.168.202.109@tcp130: rc = -75 [10218.566169] Lustre: 464863:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10221.282586] LNetError: 461070:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10221.301203] LNetError: Protocol error connecting to 192.168.202.109@tcp122 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10223.240151] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10228.558154] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10235.079736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10235.347571] Lustre: lustre-OST0000: new disk, initializing [10235.354821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10235.361657] Lustre: 465604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10235.439365] Lustre: 465609:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.109@tcp: can't add NID 192.168.202.109@tcp130: rc = -75 [10235.445364] Lustre: 465609:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [10235.490174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10237.665375] LNetError: 461069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10237.680167] LNetError: 461069:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [10237.690262] LNetError: Protocol error connecting to 192.168.202.109@tcp120 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10237.703736] LNetError: Skipped 10 previous similar messages [10241.156491] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10245.191505] Lustre: 466121:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.109@tcp: can't add NID 192.168.202.109@tcp130: rc = -75 [10245.204558] Lustre: 466121:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [10245.323764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10245.349414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10245.500202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10249.985196] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10251.104940] LNet: 1 local NIs in recovery (showing 1): 192.168.202.109@tcp123 [10260.974638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10260.981703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10260.985180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10260.995098] LustreError: Skipped 1 previous similar message [10261.023262] Lustre: Skipped 1 previous similar message [10264.583985] Lustre: server umount lustre-OST0000 complete [10270.187860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10270.203699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.229633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10270.241877] Lustre: Skipped 1 previous similar message [10270.433585] LNetError: 461068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.109 [10270.451155] LNetError: 461068:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [10270.463906] LNetError: Protocol error connecting to 192.168.202.109@tcp129 at host 192.168.202.109:988: is it running a compatible version of Lustre? [10270.471267] LNetError: Skipped 16 previous similar messages [10271.207457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.730785] Lustre: server umount lustre-MDT0000 complete [10278.794964] LustreError: 463623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788578515 with bad export cookie 15396386118813225444 [10278.807108] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10278.811242] LustreError: 463623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10279.561894] Lustre: server umount lustre-MDT0001 complete [10292.641622] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [10295.929497] Key type lgssc unregistered [10296.463797] LNet: 467517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10296.470318] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10296.475941] LNet: Removed LNI 192.168.202.109@tcp [10297.504116] LNet: Removed LNI 192.168.202.109@tcp100 [10297.505732] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10297.518131] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10297.534717] LNet: Removed LNI 192.168.202.109@tcp101 [10298.592383] LNet: Removed LNI 192.168.202.109@tcp102 [10298.600979] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10298.608106] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [10298.623778] LNet: Removed LNI 192.168.202.109@tcp103 [10299.680262] LNet: Removed LNI 192.168.202.109@tcp104 [10300.704264] LNet: Removed LNI 192.168.202.109@tcp105 [10300.707426] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10300.713472] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [10300.729366] LNet: Removed LNI 192.168.202.109@tcp106 [10301.795908] LNet: Removed LNI 192.168.202.109@tcp107 [10302.816208] LNet: Removed LNI 192.168.202.109@tcp108 [10303.840284] LNet: Removed LNI 192.168.202.109@tcp109 [10304.864310] LNet: Removed LNI 192.168.202.109@tcp110 [10304.866782] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10304.877240] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [10304.884261] LNet: Removed LNI 192.168.202.109@tcp111 [10305.889166] LNet: Removed LNI 192.168.202.109@tcp112 [10306.912381] LNet: Removed LNI 192.168.202.109@tcp113 [10307.939195] LNet: Removed LNI 192.168.202.109@tcp114 [10309.024203] LNet: Removed LNI 192.168.202.109@tcp115 [10310.048327] LNet: Removed LNI 192.168.202.109@tcp116 [10311.072443] LNet: Removed LNI 192.168.202.109@tcp117 [10312.096219] LNet: Removed LNI 192.168.202.109@tcp118 [10313.121061] LNet: Removed LNI 192.168.202.109@tcp119 [10313.123134] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10313.144047] LNetError: 467517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [10313.160523] LNet: Removed LNI 192.168.202.109@tcp120 [10314.209063] LNet: Removed LNI 192.168.202.109@tcp121 [10315.232202] LNet: Removed LNI 192.168.202.109@tcp122 [10315.234859] LNet: Removed LNI 192.168.202.109@tcp123 [10316.258609] LNet: Removed LNI 192.168.202.109@tcp124 [10317.280234] LNet: Removed LNI 192.168.202.109@tcp125 [10318.305492] LNet: Removed LNI 192.168.202.109@tcp126 [10319.328528] LNet: Removed LNI 192.168.202.109@tcp127 [10320.354430] LNet: Removed LNI 192.168.202.109@tcp128 [10321.378723] LNet: Removed LNI 192.168.202.109@tcp129 [10321.383556] LNet: Removed LNI 192.168.202.109@tcp130 [10321.389085] LNet: Removed LNI 192.168.202.109@tcp131 [10321.404485] LNet: Removed LNI 192.168.202.109@tcp132 [10321.408851] LNet: Removed LNI 192.168.202.109@tcp133 [10321.415718] LNet: Removed LNI 192.168.202.109@tcp134 [10321.440210] LNet: Removed LNI 192.168.202.109@tcp135 [10322.428372] Key type .llcrypt unregistered [10322.430703] Key type ._llcrypt unregistered [10349.061729] Key type ._llcrypt registered [10349.063085] Key type .llcrypt registered [10349.243834] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [10368.410798] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [10369.987976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10370.137206] alg: No test for adler32 (adler32-zlib) [10371.427783] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [10371.703592] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [10373.432409] Key type lgssc registered [10375.118747] Lustre: Echo OBD driver; http://www.lustre.org/ [10414.759354] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:24:10 (1788578650) [10433.157846] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [10436.304632] Key type lgssc unregistered [10436.695851] LNet: 471602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10436.709156] LNetError: 471602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10436.740588] LNet: Removed LNI 192.168.202.109@tcp [10437.617507] Key type .llcrypt unregistered [10437.626265] Key type ._llcrypt unregistered [10454.868730] Key type ._llcrypt registered [10454.876518] Key type .llcrypt registered [10454.992708] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [10456.503740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10456.547124] alg: No test for adler32 (adler32-zlib) [10457.604162] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [10457.821875] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [10459.544337] Key type lgssc registered [10460.998825] Lustre: Echo OBD driver; http://www.lustre.org/ [10472.672270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10472.731269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.210279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10474.249963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10474.389977] Lustre: lustre-MDT0000: new disk, initializing [10474.507726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10474.528844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10479.552695] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10490.630853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.769953] Lustre: 472811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10490.818494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10490.821467] Lustre: Skipped 1 previous similar message [10490.896239] Lustre: lustre-MDT0001: new disk, initializing [10490.990428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10491.016825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10491.028821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10496.334482] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10506.939491] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10514.310684] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.372585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.691740] Lustre: lustre-OST0000: new disk, initializing [10521.696963] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10521.712469] Lustre: 474782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10521.831982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10528.104128] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10531.971671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10531.992415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10532.177888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10537.926644] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10543.485844] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing lnet_if_list [10550.840559] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10550.887634] LNet: Added LNI 192.168.202.109@tcp10 [8/256/0/180] [10559.750915] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10559.811845] LNet: Added LNI 192.168.202.109@tcp11 [8/256/0/180] [10565.959646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10566.005931] LNet: Added LNI 192.168.202.109@tcp12 [8/256/0/180] [10571.651321] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10571.693959] LNet: Added LNI 192.168.202.109@tcp13 [8/256/0/180] [10577.394952] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10577.484899] LNet: Added LNI 192.168.202.109@tcp14 [8/256/0/180] [10583.213167] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10583.254718] LNet: Added LNI 192.168.202.109@tcp15 [8/256/0/180] [10590.036756] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10590.104779] LNet: Added LNI 192.168.202.109@tcp16 [8/256/0/180] [10598.547806] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10598.597440] LNet: Added LNI 192.168.202.109@tcp17 [8/256/0/180] [10604.089300] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10604.122837] LNet: Added LNI 192.168.202.109@tcp18 [8/256/0/180] [10609.985513] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10610.017576] LNet: Added LNI 192.168.202.109@tcp19 [8/256/0/180] [10616.071558] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10616.098286] LNet: Added LNI 192.168.202.109@tcp20 [8/256/0/180] [10624.790113] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10624.831594] LNet: Added LNI 192.168.202.109@tcp21 [8/256/0/180] [10632.344524] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10632.379443] LNet: Added LNI 192.168.202.109@tcp22 [8/256/0/180] [10639.393921] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10639.424690] LNet: Added LNI 192.168.202.109@tcp23 [8/256/0/180] [10645.173644] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10645.216832] LNet: Added LNI 192.168.202.109@tcp24 [8/256/0/180] [10652.741631] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10652.769406] LNet: Added LNI 192.168.202.109@tcp25 [8/256/0/180] [10660.270989] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10660.342138] LNet: Added LNI 192.168.202.109@tcp26 [8/256/0/180] [10666.842526] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10666.871227] LNet: Added LNI 192.168.202.109@tcp27 [8/256/0/180] [10673.337534] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10673.382978] LNet: Added LNI 192.168.202.109@tcp28 [8/256/0/180] [10680.786459] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10680.861580] LNet: Added LNI 192.168.202.109@tcp29 [8/256/0/180] [10687.553579] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10687.603286] LNet: Added LNI 192.168.202.109@tcp30 [8/256/0/180] [10694.787664] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10694.812267] LNet: Added LNI 192.168.202.109@tcp31 [8/256/0/180] [10702.224326] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10702.277759] LNet: Added LNI 192.168.202.109@tcp32 [8/256/0/180] [10709.763957] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10709.791368] LNet: Added LNI 192.168.202.109@tcp33 [8/256/0/180] [10718.096822] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10718.139963] LNet: Added LNI 192.168.202.109@tcp34 [8/256/0/180] [10726.562367] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10726.605667] LNet: Added LNI 192.168.202.109@tcp35 [8/256/0/180] [10734.994111] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10735.042830] LNet: Added LNI 192.168.202.109@tcp36 [8/256/0/180] [10744.089680] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10744.129683] LNet: Added LNI 192.168.202.109@tcp37 [8/256/0/180] [10755.096608] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10755.155345] LNet: Added LNI 192.168.202.109@tcp38 [8/256/0/180] [10763.382828] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10763.447391] LNet: Added LNI 192.168.202.109@tcp39 [8/256/0/180] [10773.371123] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10773.415989] LNet: Added LNI 192.168.202.109@tcp40 [8/256/0/180] [10783.347390] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10783.378226] LNet: Added LNI 192.168.202.109@tcp41 [8/256/0/180] [10791.619642] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10791.649032] LNet: Added LNI 192.168.202.109@tcp42 [8/256/0/180] [10799.772159] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10799.821595] LNet: Added LNI 192.168.202.109@tcp43 [8/256/0/180] [10809.307636] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10809.344688] LNet: Added LNI 192.168.202.109@tcp44 [8/256/0/180] [10818.578379] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10818.622645] LNet: Added LNI 192.168.202.109@tcp45 [8/256/0/180] [10827.704692] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10827.743655] LNet: Added LNI 192.168.202.109@tcp46 [8/256/0/180] [10837.640707] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10837.701336] LNet: Added LNI 192.168.202.109@tcp47 [8/256/0/180] [10851.191311] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10851.283568] LNet: Added LNI 192.168.202.109@tcp48 [8/256/0/180] [10867.026511] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10867.078168] LNet: Added LNI 192.168.202.109@tcp49 [8/256/0/180] [10879.686557] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10879.745619] LNet: Added LNI 192.168.202.109@tcp50 [8/256/0/180] [10900.457873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10900.468214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10900.488939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10904.037792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10904.043208] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.156903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10909.164362] Lustre: Skipped 4 previous similar messages [10914.273830] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10914.281755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10914.296234] Lustre: Skipped 2 previous similar messages [10914.550509] Lustre: server umount lustre-MDT0000 complete [10918.293914] LustreError: 472805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579155 with bad export cookie 4311555174075648780 [10918.297683] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.311142] LustreError: 472805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10918.653191] Lustre: server umount lustre-MDT0001 complete [10935.776490] Lustre: 472083:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579156/real 1788579156] req@0000000094ffb28d x1875460692837248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579172 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10935.814274] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10935.831079] Lustre: Skipped 1 previous similar message [10938.945805] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [10939.945748] Lustre: 472083:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579160/real 1788579160] req@0000000014abc0a8 x1875460692837376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10943.968338] Lustre: 472085:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579165/real 1788579165] req@0000000024417b71 x1875460692837760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10943.994148] Lustre: 472085:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10948.757896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.088461] Lustre: 472082:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788579170/real 1788579170] req@0000000088d16067 x1875460692837888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788579186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10963.488758] LustreError: 472081:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec798342 x1875460692838656/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10963.920731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10968.136953] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10968.995753] LustreError: 474776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.006195] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10969.034735] LustreError: 474776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10974.188915] LustreError: 474775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10978.634969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.082433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10980.085879] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10984.273886] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [10992.894814] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11000.300881] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11025.378716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11025.380123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.394418] LustreError: Skipped 1 previous similar message [11025.427343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11025.432687] Lustre: Skipped 1 previous similar message [11036.665991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.682404] Lustre: Skipped 4 previous similar messages [11038.176170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11038.741212] Lustre: server umount lustre-OST0000 complete [11045.866193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11045.878075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11045.892926] Lustre: Skipped 1 previous similar message [11052.004253] LustreError: 499038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11052.026438] LustreError: 499038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11053.065978] Lustre: server umount lustre-MDT0000 complete [11057.126671] LustreError: 499032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11057.167658] LustreError: 499032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11057.984855] LustreError: 472804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579295 with bad export cookie 4311555174075651475 [11057.990621] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11058.010615] LustreError: 472804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11058.461075] Lustre: server umount lustre-MDT0001 complete [11070.843335] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [11074.520759] Key type lgssc unregistered [11075.011123] LNet: 501838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11075.022290] LNetError: 501838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11075.033374] LNet: Removed LNI 192.168.202.109@tcp [11075.038226] LNet: Removed LNI 192.168.202.109@tcp10 [11075.046580] LNet: Removed LNI 192.168.202.109@tcp11 [11075.052113] LNet: Removed LNI 192.168.202.109@tcp12 [11075.056780] LNet: Removed LNI 192.168.202.109@tcp13 [11075.061851] LNet: Removed LNI 192.168.202.109@tcp14 [11075.065913] LNet: Removed LNI 192.168.202.109@tcp15 [11075.070915] LNet: Removed LNI 192.168.202.109@tcp16 [11075.075758] LNet: Removed LNI 192.168.202.109@tcp17 [11075.079921] LNet: Removed LNI 192.168.202.109@tcp18 [11075.085219] LNet: Removed LNI 192.168.202.109@tcp19 [11075.089424] LNet: Removed LNI 192.168.202.109@tcp20 [11075.093433] LNet: Removed LNI 192.168.202.109@tcp21 [11075.097391] LNet: Removed LNI 192.168.202.109@tcp22 [11075.102392] LNet: Removed LNI 192.168.202.109@tcp23 [11075.106881] LNet: Removed LNI 192.168.202.109@tcp24 [11075.111470] LNet: Removed LNI 192.168.202.109@tcp25 [11075.116372] LNet: Removed LNI 192.168.202.109@tcp26 [11075.119950] LNet: Removed LNI 192.168.202.109@tcp27 [11075.124324] LNet: Removed LNI 192.168.202.109@tcp28 [11075.128365] LNet: Removed LNI 192.168.202.109@tcp29 [11075.132215] LNet: Removed LNI 192.168.202.109@tcp30 [11075.138381] LNet: Removed LNI 192.168.202.109@tcp31 [11075.144359] LNet: Removed LNI 192.168.202.109@tcp32 [11075.149334] LNet: Removed LNI 192.168.202.109@tcp33 [11075.153025] LNet: Removed LNI 192.168.202.109@tcp34 [11075.156348] LNet: Removed LNI 192.168.202.109@tcp35 [11075.160654] LNet: Removed LNI 192.168.202.109@tcp36 [11075.163293] LNet: Removed LNI 192.168.202.109@tcp37 [11075.166316] LNet: Removed LNI 192.168.202.109@tcp38 [11075.171230] LNet: Removed LNI 192.168.202.109@tcp39 [11075.175342] LNet: Removed LNI 192.168.202.109@tcp40 [11075.180371] LNet: Removed LNI 192.168.202.109@tcp41 [11075.185242] LNet: Removed LNI 192.168.202.109@tcp42 [11075.189761] LNet: Removed LNI 192.168.202.109@tcp43 [11075.193376] LNet: Removed LNI 192.168.202.109@tcp44 [11075.198204] LNet: Removed LNI 192.168.202.109@tcp45 [11075.202386] LNet: Removed LNI 192.168.202.109@tcp46 [11075.205548] LNet: Removed LNI 192.168.202.109@tcp47 [11075.210093] LNet: Removed LNI 192.168.202.109@tcp48 [11075.211766] LNet: Removed LNI 192.168.202.109@tcp49 [11075.223721] LNet: Removed LNI 192.168.202.109@tcp50 [11076.146964] Key type .llcrypt unregistered [11076.151934] Key type ._llcrypt unregistered [11097.788981] Key type ._llcrypt registered [11097.791632] Key type .llcrypt registered [11097.890646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [11112.593129] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [11114.126967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11114.162733] alg: No test for adler32 (adler32-zlib) [11115.382164] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11115.742167] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [11117.416219] Key type lgssc registered [11118.761722] Lustre: Echo OBD driver; http://www.lustre.org/ [11164.152288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11164.219631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11165.504890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11165.561350] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11165.644714] Lustre: lustre-MDT0000: new disk, initializing [11165.749403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11165.770933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11171.639198] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11184.500948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11184.735884] Lustre: 505030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11184.839392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11184.858451] Lustre: Skipped 1 previous similar message [11184.979677] Lustre: lustre-MDT0001: new disk, initializing [11185.165373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11185.226512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11185.236686] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11190.662040] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11204.315292] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11212.762362] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11221.568391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11221.855945] Lustre: *** cfs_fail_loc=90f, val=0*** [11222.049818] Lustre: lustre-OST0000: new disk, initializing [11222.061390] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11222.085281] Lustre: 507052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11222.290482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11228.308284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11228.324652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11228.520156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11231.188755] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11241.802675] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11254.256588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11254.262426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11254.264078] LustreError: Skipped 1 previous similar message [11254.269500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11254.277741] Lustre: Skipped 1 previous similar message [11254.311885] Lustre: Skipped 1 previous similar message [11256.819567] Lustre: server umount lustre-OST0000 complete [11261.923155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11261.945792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.969028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.482971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.485230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11269.604048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.614337] Lustre: Skipped 2 previous similar messages [11274.727297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11274.746587] Lustre: Skipped 1 previous similar message [11276.256480] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11276.585795] Lustre: server umount lustre-MDT0000 complete [11282.438137] LustreError: 505836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579519 with bad export cookie 16826825283813920642 [11282.448659] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11282.453039] LustreError: 505836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11283.233832] Lustre: server umount lustre-MDT0001 complete [11299.725345] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [11304.367467] Key type lgssc unregistered [11304.977326] LNet: 508979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11304.988798] LNetError: 508979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11305.020706] LNet: Removed LNI 192.168.202.109@tcp [11306.568124] Key type .llcrypt unregistered [11306.570304] Key type ._llcrypt unregistered [11333.170559] Key type ._llcrypt registered [11333.179638] Key type .llcrypt registered [11333.380442] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [11335.275110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11335.565460] alg: No test for adler32 (adler32-zlib) [11337.228142] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11337.650636] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [11339.394225] Key type lgssc registered [11340.857613] Lustre: Echo OBD driver; http://www.lustre.org/ [11348.837685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11348.877058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.409431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11356.110851] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11360.744504] LustreError: 510247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11360.769679] LustreError: 510247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11365.865779] LustreError: 510246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.032664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.480560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11372.042816] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11380.081594] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11385.747771] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11392.862922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11393.222928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11400.823797] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11409.142432] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11423.761262] loop0: detected capacity change from 0 to 204800000 [11424.154760] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11424.168657] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11424.190653] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11430.908315] loop0: detected capacity change from 0 to 204800000 [11431.197542] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11431.207229] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11431.217638] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11437.530791] loop0: detected capacity change from 0 to 204800000 [11443.933886] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11444.172433] Lustre: fs15246-OST0000: new disk, initializing [11444.178033] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [11444.189708] Lustre: 513841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11444.260308] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11452.708986] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11463.347761] loop1: detected capacity change from 0 to 204800000 [11469.628651] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.885989] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [11469.978099] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [11470.153194] Lustre: fs15246-MDT0000: new disk, initializing [11470.375418] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11470.420091] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11470.598111] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11471.650598] Lustre: 514964:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004ce776c9 x1875461615439232/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788579724 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [11475.645885] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11476.717803] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11476.736830] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11476.796771] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11506.180963] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.210554] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [11508.305950] Lustre: server umount fs15246-MDT0000 complete [11511.301327] LustreError: 510241:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.399612] LustreError: 510243:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11521.507204] LustreError: 512616:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.467080] Lustre: server umount fs15246-OST0000 complete [11541.992837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11542.037242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11542.051624] Lustre: Skipped 1 previous similar message [11547.065358] Lustre: server umount lustre-OST0000 complete [11552.226632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.242389] Lustre: Skipped 1 previous similar message [11552.252354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11557.346754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11557.358517] Lustre: Skipped 2 previous similar messages [11558.762397] Lustre: server umount lustre-MDT0000 complete [11562.479835] LustreError: 510241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11562.525550] LustreError: 510241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11563.378479] LustreError: 510981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788579800 with bad export cookie 3854759410798419813 [11563.386268] LustreError: 510981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11563.389523] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11564.046819] Lustre: server umount lustre-MDT0001 complete [11578.218826] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [11581.889312] Key type lgssc unregistered [11582.463063] LNet: 518062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11582.483737] LNetError: 518062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11582.510247] LNet: Removed LNI 192.168.202.109@tcp [11584.104443] Key type .llcrypt unregistered [11584.108780] Key type ._llcrypt unregistered [11612.823622] Key type ._llcrypt registered [11612.827491] Key type .llcrypt registered [11612.974927] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [11614.698226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11614.706242] alg: No test for adler32 (adler32-zlib) [11615.797722] Lustre: Lustre: Build Version: 2.17.58_2_gfa0a981 [11616.031426] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [11617.657833] Key type lgssc registered [11618.965645] Lustre: Echo OBD driver; http://www.lustre.org/ [11669.082987] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [11682.912792] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [11731.015046] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11731.034938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11732.267644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11732.313075] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11732.413450] Lustre: lustre-MDT0000: new disk, initializing [11732.484912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11732.500151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11737.700105] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11749.456439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11749.588440] Lustre: 522534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11749.663433] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11749.667818] Lustre: Skipped 1 previous similar message [11749.787860] Lustre: lustre-MDT0001: new disk, initializing [11749.926768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11749.994833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11750.016436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11755.045841] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11765.299619] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11770.689157] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11778.130043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11778.763905] Lustre: lustre-OST0000: new disk, initializing [11778.777049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11778.788562] Lustre: 524501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11778.924053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11785.355354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11785.386725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11785.613425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11786.993232] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [11798.757594] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11804.753919] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11805.040702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11809.823308] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11810.139186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11816.442091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11816.445408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11816.452930] LustreError: Skipped 1 previous similar message [11816.461434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11816.477161] Lustre: Skipped 1 previous similar message [11816.505789] Lustre: Skipped 1 previous similar message [11821.545297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11821.553394] Lustre: Skipped 1 previous similar message [11826.666956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11826.690485] Lustre: Skipped 1 previous similar message [11827.168176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11827.489869] Lustre: server umount lustre-OST0000 complete [11836.901547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11836.902682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11836.915224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11836.943802] Lustre: Skipped 1 previous similar message [11838.425538] Lustre: server umount lustre-MDT0000 complete [11843.454184] LustreError: 522525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788580080 with bad export cookie 3163486964490209279 [11843.458315] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11843.468424] LustreError: 522525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11844.466958] Lustre: server umount lustre-MDT0001 complete [11874.060543] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [11886.631640] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [11926.702562] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 11582 sec ========= 23:49:21 (1788580161) [11929.024581] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:49:24 (1788580164) === [11933.380664] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:49:29 (1788580169) === [11987.817111] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [11990.372484] Key type lgssc unregistered [11990.728121] LNet: 531155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11990.741887] LNetError: 531155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11991.782903] LNet: Removed LNI 192.168.202.109@tcp [11993.183468] Key type .llcrypt unregistered [11993.185697] Key type ._llcrypt unregistered