[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504962098 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003116] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013091] *** VALIDATE selinux *** [ 0.022640] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027443] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030133] *** VALIDATE tmpfs *** [ 0.031346] *** VALIDATE proc *** [ 0.032209] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034211] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036038] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042499] debug: unmapping init [mem 0xffffffff97459000-0xffffffff97460fff] [ 0.044831] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045738] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.047998] ... generic registers: 4 [ 0.048017] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052309] rcu: Hierarchical SRCU implementation. [ 0.054472] smp: Bringing up secondary CPUs ... [ 0.055538] x86: Booting SMP configuration: [ 0.056034] .... node #0, CPUs: #1 #2 #3 [ 0.059600] smp: Brought up 1 node, 4 CPUs [ 0.061020] smpboot: Max logical packages: 1 [ 0.062022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141021] node 0 deferred pages initialised in 77ms [ 0.144470] devtmpfs: initialized [ 0.145201] x86/mm: Memory block size: 128MB [ 0.147106] gcov: version magic: 0x41383552 [ 0.151319] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156336] pinctrl core: initialized pinctrl subsystem [ 0.158210] [ 0.158968] ************************************************************* [ 0.163018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166014] ** ** [ 0.169014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172019] ** ** [ 0.175016] ** This means that this kernel is built to expose internal ** [ 0.179015] ** IOMMU data structures, which may compromise security on ** [ 0.182021] ** your system. ** [ 0.185018] ** ** [ 0.188020] ** If you see this message and you are not debugging the ** [ 0.191045] ** kernel, report this immediately to your vendor! ** [ 0.194015] ** ** [ 0.197015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200015] ************************************************************* [ 0.204705] NET: Registered protocol family 16 [ 0.207559] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.210077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216034] cpuidle: using governor menu [ 0.217703] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221482] PCI: Using configuration type 1 for base access [ 0.223123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240034] cryptd: max_cpu_qlen set to 1000 [ 0.242571] ACPI: Added _OSI(Module Device) [ 0.243014] ACPI: Added _OSI(Processor Device) [ 0.245017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253536] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260587] ACPI: Interpreter enabled [ 0.262070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264032] ACPI: Using IOAPIC for interrupt routing [ 0.266125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.269436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279158] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.282036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288104] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293519] acpiphp: Slot [2] registered [ 0.295144] acpiphp: Slot [5] registered [ 0.297091] acpiphp: Slot [6] registered [ 0.299147] acpiphp: Slot [7] registered [ 0.300133] acpiphp: Slot [8] registered [ 0.302124] acpiphp: Slot [9] registered [ 0.303153] acpiphp: Slot [10] registered [ 0.305251] acpiphp: Slot [3] registered [ 0.306094] acpiphp: Slot [4] registered [ 0.308100] acpiphp: Slot [11] registered [ 0.309089] acpiphp: Slot [12] registered [ 0.311102] acpiphp: Slot [13] registered [ 0.312122] acpiphp: Slot [14] registered [ 0.314102] acpiphp: Slot [15] registered [ 0.315090] acpiphp: Slot [16] registered [ 0.316090] acpiphp: Slot [17] registered [ 0.317114] acpiphp: Slot [18] registered [ 0.318150] acpiphp: Slot [19] registered [ 0.320134] acpiphp: Slot [20] registered [ 0.321188] acpiphp: Slot [21] registered [ 0.322104] acpiphp: Slot [22] registered [ 0.324095] acpiphp: Slot [23] registered [ 0.325176] acpiphp: Slot [24] registered [ 0.327081] acpiphp: Slot [25] registered [ 0.328107] acpiphp: Slot [26] registered [ 0.330103] acpiphp: Slot [27] registered [ 0.331127] acpiphp: Slot [28] registered [ 0.333117] acpiphp: Slot [29] registered [ 0.335107] acpiphp: Slot [30] registered [ 0.336120] acpiphp: Slot [31] registered [ 0.337053] PCI host bridge to bus 0000:00 [ 0.339019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.341023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357578] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361307] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371862] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.377043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.382016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387722] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391145] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397768] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.416833] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.422025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428387] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.438020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.445015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472065] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.480014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.539015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.560018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.569888] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.576014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.584016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.603020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.615896] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.625018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.631019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.650014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662511] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.677016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.694016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.705386] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.708417] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.711384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.713389] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.716250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.722114] iommu: Default domain type: Passthrough [ 0.725400] SCSI subsystem initialized [ 0.726316] ACPI: bus type USB registered [ 0.728108] usbcore: registered new interface driver usbfs [ 0.731095] usbcore: registered new interface driver hub [ 0.733079] usbcore: registered new device driver usb [ 0.734155] pps_core: LinuxPPS API ver. 1 registered [ 0.736011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740093] PTP clock support registered [ 0.742077] EDAC MC: Ver: 3.0.0 [ 0.744009] PCI: Using ACPI for IRQ routing [ 0.746052] NetLabel: Initializing [ 0.747010] NetLabel: domain hash size = 128 [ 0.748010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.750068] NetLabel: unlabeled traffic allowed by default [ 0.752136] vgaarb: loaded [ 0.753244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.765037] clocksource: Switched to clocksource kvm-clock [ 0.872827] VFS: Disk quotas dquot_6.6.0 [ 0.874914] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.877805] *** VALIDATE ramfs *** [ 0.879105] *** VALIDATE hugetlbfs *** [ 0.881075] pnp: PnP ACPI init [ 0.883667] pnp: PnP ACPI: found 6 devices [ 0.903346] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.907210] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.910037] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.912518] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.915053] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.917590] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.920748] NET: Registered protocol family 2 [ 0.923413] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.928431] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932282] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937610] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.940724] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943784] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.947109] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.950132] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.953042] NET: Registered protocol family 1 [ 0.955283] RPC: Registered named UNIX socket transport module. [ 0.957756] RPC: Registered udp transport module. [ 0.959752] RPC: Registered tcp transport module. [ 0.961544] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.963520] NET: Registered protocol family 44 [ 0.965322] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967841] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970261] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.973068] PCI: CLS 0 bytes, default 64 [ 0.975594] Unpacking initramfs... [ 2.378311] debug: unmapping init [mem 0xffff8da63cc54000-0xffff8da63ffbffff] [ 2.382558] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.384942] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.388242] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.876335] Initialise system trusted keyrings [ 2.878100] Key type blacklist registered [ 2.880451] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.891155] zbud: loaded [ 2.895794] *** VALIDATE nfs *** [ 2.897164] *** VALIDATE nfs4 *** [ 2.898981] pstore: using deflate compression [ 2.902898] Platform Keyring initialized [ 3.010838] NET: Registered protocol family 38 [ 3.013449] Key type asymmetric registered [ 3.015228] Asymmetric key parser 'x509' registered [ 3.017577] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.020984] io scheduler mq-deadline registered [ 3.022301] io scheduler kyber registered [ 3.023735] io scheduler bfq registered [ 3.025702] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.029096] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.031508] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.033761] ACPI: Power Button [PWRF] [ 3.038754] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.044789] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.068033] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.080627] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.107340] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.135134] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.166809] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.172799] Non-volatile memory driver v1.3 [ 3.174238] Linux agpgart interface v0.103 [ 3.202282] virtio_blk virtio1: [vda] 146616 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.206485] vda: detected capacity change from 0 to 75067392 [ 3.222843] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.225671] vdb: detected capacity change from 0 to 1073741824 [ 3.241018] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.244028] vdc: detected capacity change from 0 to 2621440000 [ 3.257669] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.260891] vdd: detected capacity change from 0 to 2621440000 [ 3.276934] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.280142] vde: detected capacity change from 0 to 4294967296 [ 3.295188] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.298283] vdf: detected capacity change from 0 to 4294967296 [ 3.307868] libphy: Fixed MDIO Bus: probed [ 3.316325] usbcore: registered new interface driver usbserial_generic [ 3.318233] usbserial: USB Serial support registered for generic [ 3.320415] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.325986] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.328128] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.331720] mousedev: PS/2 mouse device common for all mice [ 3.334476] rtc_cmos 00:05: RTC can wake from S4 [ 3.337397] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.337696] rtc_cmos 00:05: registered as rtc0 [ 3.342959] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.346253] intel_pstate: CPU model not supported [ 3.349193] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.351380] hid: raw HID events driver (C) Jiri Kosina [ 3.355289] usbcore: registered new interface driver usbhid [ 3.357874] usbhid: USB HID core driver [ 3.359731] drop_monitor: Initializing network drop monitor service [ 3.359774] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.362496] Initializing XFRM netlink socket [ 3.368096] NET: Registered protocol family 10 [ 3.370986] Segment Routing with IPv6 [ 3.372730] NET: Registered protocol family 17 [ 3.375788] mpls_gso: MPLS GSO support [ 3.381976] RAS: Correctable Errors collector initialized. [ 3.383787] AVX version of gcm_enc/dec engaged. [ 3.385311] AES CTR mode by8 optimization enabled [ 3.456065] sched_clock: Marking stable (3456027137, 0)->(4380414911, -924387774) [ 3.458813] registered taskstats version 1 [ 3.460505] Loading compiled-in X.509 certificates [ 3.462668] zswap: loaded using pool lzo/zbud [ 3.484282] Key type big_key registered [ 3.495125] Key type encrypted registered [ 3.496533] ima: No TPM chip found, activating TPM-bypass! [ 3.498465] ima: Allocated hash algorithm: sha1 [ 3.499870] ima: No architecture policies found [ 3.501245] evm: Initialising EVM extended attributes: [ 3.502744] evm: security.selinux [ 3.503783] evm: security.ima [ 3.504683] evm: security.capability [ 3.505736] evm: HMAC attrs: 0x1 [ 3.507805] rtc_cmos 00:05: setting system clock to 2026-08-26 22:49:35 UTC (1787784575) [ 3.515144] debug: unmapping init [mem 0xffffffff98403000-0xffffffff985fffff] [ 3.517680] debug: unmapping init [mem 0xffffffff97182000-0xffffffff97458fff] [ 3.526083] Write protecting the kernel read-only data: 28672k [ 3.529697] debug: unmapping init [mem 0xffffffff95803000-0xffffffff959fffff] [ 3.533343] debug: unmapping init [mem 0xffffffff96114000-0xffffffff961fffff] [ 3.565873] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.572761] systemd[1]: Detected virtualization kvm. [ 3.574503] systemd[1]: Detected architecture x86-64. [ 3.576024] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.600986] systemd[1]: No hostname configured. [ 3.602306] systemd[1]: Set hostname to . [ 3.603914] random: systemd: uninitialized urandom read (16 bytes read) [ 3.605875] systemd[1]: Initializing machine ID from random generator. [ 3.631242] random: ln: uninitialized urandom read (6 bytes read) [ 3.734496] random: systemd: uninitialized urandom read (16 bytes read) [ 3.736898] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.741449] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.744643] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.293330] device-mapper: uevent: version 1.0.3 [ 4.296640] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [[ 4.918654] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.967757] virtio_net virtio0 ens2: renamed from eth0 [ 5.029311] scsi host0: ata_piix [ 5.046173] scsi host1: ata_piix [ 5.048032] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.050649] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.325426] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.012109] random: crng init done [ 10.013573] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.325957] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.470444] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.726217] SELinux: Disabled at runtime. [ 11.786686] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.797146] systemd[1]: Detected virtualization kvm. [ 11.799122] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.306292] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.309641] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.315462] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.319707] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.323217] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.332303] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.338670] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Started Create list of required [ 12.556654] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.841838] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.145067] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.155265] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.260323] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.275935] EDAC sbridge: Ver: 1.1.2 [ 15.135545] Key type dns_resolver registered [ 15.443676] NFS: Registering the id_resolver key type [ 15.445550] Key type id_resolver registered [ 15.447055] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg422-server login: [ 42.027584] libcfs: loading out-of-tree module taints kernel. [ 42.052257] Key type ._llcrypt registered [ 42.054572] Key type .llcrypt registered [ 42.113791] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 52.185992] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 53.297477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.317653] alg: No test for adler32 (adler32-zlib) [ 54.920532] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 56.025727] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 57.857876] Key type lgssc registered [ 60.365533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.775963] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.971829] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 142.165838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.202556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.628837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.665694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.772912] Lustre: lustre-MDT0000: new disk, initializing [ 143.917898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.940254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.920818] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.492337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.642355] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.673251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.677656] Lustre: Skipped 1 previous similar message [ 163.830898] Lustre: lustre-MDT0001: new disk, initializing [ 163.927173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 163.973202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 163.992637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.764228] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.273715] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.552670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.916623] Lustre: lustre-OST0000: new disk, initializing [ 186.923494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.933110] Lustre: 8426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 187.069929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.439545] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.310156] hrtimer: interrupt took 2167160 ns [ 196.665282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.679880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.758661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 211.578384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.779439] Lustre: lustre-OST0001: new disk, initializing [ 211.782747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.789612] Lustre: 9499:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.869723] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.143743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 217.151754] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 217.216394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.094510] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.176655] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.898028] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.849852] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.632435] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing yml_node [ 257.064836] Lustre: DEBUG MARKER: Client: 2.17.57.54 [ 259.859212] Lustre: DEBUG MARKER: MDS: 2.17.57.54 [ 262.247275] Lustre: DEBUG MARKER: OSS: 2.17.57.54 [ 264.185498] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 26 18:53:55 EDT 2026 [ 282.501770] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 284.494514] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.050766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.067179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.099282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.097975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.099291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.219347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.233756] Lustre: Skipped 6 previous similar messages [ 314.336750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.349185] Lustre: Skipped 3 previous similar messages [ 315.359133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.534670] Lustre: server umount lustre-MDT0000 complete [ 320.399337] LustreError: 10317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787784892 with bad export cookie 9388462297818450287 [ 320.404765] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.415802] LustreError: 10317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 324.580110] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.580600] LustreError: 6495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.580612] LustreError: 6495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 324.584312] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.600201] Lustre: Skipped 3 previous similar messages [ 324.645975] Lustre: Skipped 1 previous similar message [ 325.605322] LustreError: 10075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.642874] LustreError: 10075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 326.862367] Lustre: server umount lustre-MDT0001 complete [ 332.112237] Lustre: server umount lustre-OST0000 complete [ 336.980521] Lustre: server umount lustre-OST0001 complete [ 344.316942] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 355.482992] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 393.983373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.216170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 394.248962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 394.329789] Lustre: lustre-MDT0000: new disk, initializing [ 394.408684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.433972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.802962] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 413.088590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.291441] Lustre: 15058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.373487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.384410] Lustre: Skipped 1 previous similar message [ 413.560559] Lustre: lustre-MDT0001: new disk, initializing [ 413.687687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 413.729621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 413.750419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.631735] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 428.630789] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 434.236342] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 440.972063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.410188] Lustre: lustre-OST0000: new disk, initializing [ 441.415454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.422700] Lustre: 17028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.495953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.953475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 442.970858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.066828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.621426] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 456.643290] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 460.851263] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.047546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 464.750689] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 464.917409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 468.970215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 468.971503] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.986357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 469.002838] Lustre: Skipped 1 previous similar message [ 469.017247] Lustre: Skipped 3 previous similar messages [ 481.247216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 481.478077] Lustre: server umount lustre-OST0000 complete [ 485.344226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 485.352553] LustreError: Skipped 1 previous similar message [ 485.355606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.365482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.369196] Lustre: Skipped 4 previous similar messages [ 491.639951] Lustre: server umount lustre-MDT0000 complete [ 494.561147] LustreError: 15066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.579403] LustreError: 15066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 495.680452] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785067 with bad export cookie 9388462297818451827 [ 495.684058] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.691985] LustreError: 15051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 496.220228] Lustre: server umount lustre-MDT0001 complete [ 501.778612] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:57:52 (1787785072) [ 505.138186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 507.027945] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:57:58 (1787785078) [ 514.628606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.241469] LustreError: 19437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.342460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.358956] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 520.679263] LustreError: 19438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.517339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.258410] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 546.130940] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.485523] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 562.515587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.848844] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 580.770361] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 594.917757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.931216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.961164] Lustre: Skipped 1 previous similar message [ 594.965757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 594.976028] Lustre: Skipped 2 previous similar messages [ 600.095237] Lustre: server umount lustre-OST0000 complete [ 610.770809] Lustre: server umount lustre-MDT0000 complete [ 611.299873] LustreError: 21158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.337819] LustreError: 21158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 615.375613] LustreError: 19417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785187 with bad export cookie 9388462297818452709 [ 615.393714] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.399432] LustreError: 19417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 615.827583] Lustre: server umount lustre-MDT0001 complete [ 628.481275] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 632.067422] Key type lgssc unregistered [ 632.527412] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.540953] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.575941] LNet: Removed LNI 192.168.204.122@tcp [ 634.668254] Key type .llcrypt unregistered [ 634.675797] Key type ._llcrypt unregistered [ 660.665216] Key type ._llcrypt registered [ 660.669465] Key type .llcrypt registered [ 660.839724] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 662.394279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 662.462345] alg: No test for adler32 (adler32-zlib) [ 663.500817] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 663.739192] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 665.431158] Key type lgssc registered [ 666.522282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 675.253912] loop: module loaded [ 675.348702] loop0: detected capacity change from 0 to 204800000 [ 675.592682] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 675.602701] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 676.050895] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.248101] loop0: detected capacity change from 0 to 204800000 [ 683.583638] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.603331] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 683.634100] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 690.654743] loop0: detected capacity change from 0 to 204800000 [ 697.361739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.454480] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.984225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 699.053542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 699.185604] Lustre: lustre-MDT0000: new disk, initializing [ 699.342898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.404698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 706.161158] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 716.590101] loop1: detected capacity change from 0 to 204800000 [ 723.005797] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.523390] Lustre: lustre-OST07c6: new disk, initializing [ 723.527679] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 723.533783] Lustre: Skipped 1 previous similar message [ 723.542916] Lustre: 26201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 723.658454] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 731.365906] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 731.824906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 731.839741] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 731.911606] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 736.582649] Lustre: 25186:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 736.595744] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 736.602826] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 736.624354] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 736.635803] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 736.645617] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 748.364235] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 748.387154] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 748.396731] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 748.407520] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 748.423680] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 748.430640] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.740635] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 753.749926] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 753.760752] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 753.771201] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.781320] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.792570] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.797927] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 753.802982] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.807559] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 753.814747] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.818904] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.823471] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.045322] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 757.053164] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 757.059486] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 757.066074] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.071322] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 757.076048] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.080630] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 757.085789] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.090597] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 757.095425] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 757.100486] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 757.106980] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.343567] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 762.359941] Lustre: 26719:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 762.367246] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 762.372120] Lustre: 26719:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.377163] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 762.382517] Lustre: 26719:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.388105] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 762.394026] Lustre: 26719:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.399009] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 762.404208] Lustre: 26719:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 762.409109] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 762.414445] Lustre: 26719:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.835737] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 773.850219] Lustre: 25187:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 773.860816] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 773.875678] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 773.886028] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 773.896169] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 773.907410] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 773.920788] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 773.932426] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 773.947200] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 773.961772] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.971190] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.899023] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.908979] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 831.815505] Lustre: server umount lustre-OST07c6 complete [ 841.323733] Lustre: server umount lustre-MDT0000 complete [ 860.669626] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 864.074201] Key type lgssc unregistered [ 864.433233] LNet: 28765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.441255] LNetError: 28765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.456512] LNet: Removed LNI 192.168.204.122@tcp [ 865.595599] Key type .llcrypt unregistered [ 865.598428] Key type ._llcrypt unregistered [ 889.904580] Key type ._llcrypt registered [ 889.907848] Key type .llcrypt registered [ 890.053528] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 891.467974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.754692] alg: No test for adler32 (adler32-zlib) [ 892.974834] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 893.299949] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 894.992342] Key type lgssc registered [ 896.610752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 906.596160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 906.636246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.323740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.803803] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 918.497308] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.540715] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 923.621703] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 923.701241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.249907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.228737] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 938.720766] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 946.133614] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 954.914698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.313615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.753951] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 972.256430] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 978.873595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 999.402236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.430781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.431868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.470991] Lustre: Skipped 1 previous similar message [ 1004.519182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1004.542664] Lustre: Skipped 1 previous similar message [ 1009.641240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.645312] Lustre: Skipped 1 previous similar message [ 1013.215207] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1013.436220] Lustre: server umount lustre-OST0000 complete [ 1019.877793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.881835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.895044] Lustre: Skipped 2 previous similar messages [ 1024.008106] Lustre: server umount lustre-MDT0000 complete [ 1024.992760] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.008623] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1027.784885] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785599 with bad export cookie 3214168886269213146 [ 1027.796543] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1027.812154] LustreError: 30019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1028.371313] Lustre: server umount lustre-MDT0001 complete [ 1039.015738] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1041.659558] Key type lgssc unregistered [ 1042.010847] LNet: 33804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.020991] LNetError: 33804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.045940] LNet: Removed LNI 192.168.204.122@tcp [ 1042.988224] Key type .llcrypt unregistered [ 1042.990977] Key type ._llcrypt unregistered [ 1062.844679] Key type ._llcrypt registered [ 1062.851982] Key type .llcrypt registered [ 1062.956743] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1064.266953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.365571] alg: No test for adler32 (adler32-zlib) [ 1065.624596] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 1065.965319] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1067.721600] Key type lgssc registered [ 1069.109824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1078.101909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1078.153196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.707713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1084.390315] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1090.020882] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1090.037434] LustreError: 35075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1093.851878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.284274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1099.936128] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1110.203640] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1118.326923] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1127.069942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.655161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.703576] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1143.710447] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1150.205728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1165.794214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1165.795434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.805832] LustreError: Skipped 1 previous similar message [ 1165.809214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1165.837552] Lustre: Skipped 1 previous similar message [ 1170.929745] LustreError: 37480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.958562] LustreError: 37480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1171.085875] Lustre: server umount lustre-OST0000 complete [ 1176.035841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.046635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1176.050068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1176.057433] Lustre: Skipped 1 previous similar message [ 1176.080090] Lustre: Skipped 2 previous similar messages [ 1181.163065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.172586] Lustre: Skipped 1 previous similar message [ 1181.946469] Lustre: server umount lustre-MDT0000 complete [ 1186.274408] LustreError: 36562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.308481] LustreError: 36562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1186.618278] LustreError: 37880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785758 with bad export cookie 12093560097718029006 [ 1186.623864] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.632381] LustreError: 37880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1186.956920] Lustre: server umount lustre-MDT0001 complete [ 1200.233261] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1203.151841] Key type lgssc unregistered [ 1203.493493] LNet: 38636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.498705] LNetError: 38636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.520928] LNet: Removed LNI 192.168.204.122@tcp [ 1204.225365] Key type .llcrypt unregistered [ 1204.227093] Key type ._llcrypt unregistered [ 1231.482791] Key type ._llcrypt registered [ 1231.485886] Key type .llcrypt registered [ 1231.578775] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1232.845200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1233.109864] alg: No test for adler32 (adler32-zlib) [ 1234.372580] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 1234.929096] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1236.640352] Key type lgssc registered [ 1238.229897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1247.175130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1247.209616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.840650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.429707] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1258.981966] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.998996] LustreError: 40052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1263.017514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.416889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1267.849209] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1275.565315] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.641275] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1288.998713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1289.363980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.497280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1296.570092] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1305.218364] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1314.278563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1314.286107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1314.292529] Lustre: Skipped 2 previous similar messages [ 1314.304282] Lustre: Skipped 2 previous similar messages [ 1317.161342] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 1319.403606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1319.409103] Lustre: Skipped 2 previous similar messages [ 1320.725188] Lustre: server umount lustre-MDT0000 complete [ 1322.237019] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.512315] LustreError: 40053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.534617] LustreError: 40053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1324.767305] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787785896 with bad export cookie 333093711088848967 [ 1324.770512] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1324.793798] LustreError: 40034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.356379] LustreError: 40048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.394304] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.22@tcp (stopping) [ 1329.637201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.474148] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.474495] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.22@tcp (stopping) [ 1332.504451] LustreError: 40047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1332.537533] Lustre: Skipped 1 previous similar message [ 1339.359319] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1339.642432] Lustre: server umount lustre-MDT0001 complete [ 1355.199210] Lustre: server umount lustre-OST0000 complete [ 1374.064745] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1377.312514] Key type lgssc unregistered [ 1377.667873] LNet: 43919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.672832] LNetError: 43919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.695202] LNet: Removed LNI 192.168.204.122@tcp [ 1378.682725] Key type .llcrypt unregistered [ 1378.686299] Key type ._llcrypt unregistered [ 1402.091804] Key type ._llcrypt registered [ 1402.096926] Key type .llcrypt registered [ 1402.221371] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1403.463724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.515216] alg: No test for adler32 (adler32-zlib) [ 1404.730765] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 1405.019976] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1406.727811] Key type lgssc registered [ 1407.713924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.502562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.517987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.873811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.000982] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1426.401924] LustreError: 45337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.419156] LustreError: 45337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1430.465203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.735496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.419631] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1442.750776] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1448.711075] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1455.105913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1455.482608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1461.615538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1461.647858] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1468.885811] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1476.381392] Lustre: lustre-OST0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 1477.088703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1477.092508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1477.101775] LustreError: Skipped 1 previous similar message [ 1477.117493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.122284] Lustre: Skipped 1 previous similar message [ 1481.476493] Lustre: lustre-OST0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 1486.587307] Lustre: lustre-OST0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 1486.595589] Lustre: Skipped 2 previous similar messages [ 1489.375143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1489.482715] Lustre: server umount lustre-OST0000 complete [ 1507.810085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1507.823200] Lustre: Skipped 1 previous similar message [ 1507.834979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1507.839029] Lustre: Skipped 2 previous similar messages [ 1512.511467] Lustre: server umount lustre-MDT0000 complete [ 1512.930908] LustreError: 45337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.966065] LustreError: 45337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1515.920685] LustreError: 46073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786087 with bad export cookie 18342509455891520772 [ 1515.930588] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.335437] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1530.622183] Lustre: server umount lustre-MDT0001 complete [ 1542.649420] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1544.899498] Key type lgssc unregistered [ 1545.108922] LNet: 49053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.116609] LNetError: 49053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.126910] LNet: Removed LNI 192.168.204.122@tcp [ 1545.781906] Key type .llcrypt unregistered [ 1545.783411] Key type ._llcrypt unregistered [ 1571.661681] Key type ._llcrypt registered [ 1571.665775] Key type .llcrypt registered [ 1571.765169] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1573.037209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.077347] alg: No test for adler32 (adler32-zlib) [ 1574.152722] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 1574.416473] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1576.103371] Key type lgssc registered [ 1577.241269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1585.724389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1585.739888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.237800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.247629] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1597.414519] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.434048] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1599.218668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1599.539644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.447568] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1611.196967] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1617.451695] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1624.829761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1625.243688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.431812] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1632.370294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1639.242636] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1649.899332] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1653.562923] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1665.921660] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2619h17m16s [ 1668.035921] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1671.139909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1671.145367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.160826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.237704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.244740] Lustre: Skipped 4 previous similar messages [ 1674.046488] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 1674.060912] Lustre: Skipped 1 previous similar message [ 1677.318203] Lustre: server umount lustre-MDT0000 complete [ 1678.311397] LustreError: 52912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1678.348160] LustreError: 52912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1680.119676] LustreError: 50467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.820523] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1684.458259] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1684.468585] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1687.835145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1687.917510] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.142697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.196979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1689.212878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1692.926361] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1696.430891] Lustre: DEBUG MARKER: Wait for df (36766) ... [ 1698.203101] Lustre: DEBUG MARKER: done [ 1703.391954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1703.400070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.411445] Lustre: Skipped 2 previous similar messages [ 1703.419484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1704.940359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.059114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.066939] Lustre: Skipped 3 previous similar messages [ 1717.216565] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1717.537325] Lustre: server umount lustre-OST0000 complete [ 1725.411444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1725.412816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.426172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.436164] Lustre: Skipped 1 previous similar message [ 1725.445278] Lustre: Skipped 3 previous similar messages [ 1728.459483] Lustre: server umount lustre-MDT0000 complete [ 1732.058214] LustreError: 50450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786304 with bad export cookie 12082658331855036661 [ 1732.060803] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1732.071683] LustreError: 50450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1732.382778] Lustre: server umount lustre-MDT0001 complete [ 1743.310479] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1746.293550] Key type lgssc unregistered [ 1746.658385] LNet: 55404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.677494] LNetError: 55404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.713709] LNet: Removed LNI 192.168.204.122@tcp [ 1747.656695] Key type .llcrypt unregistered [ 1747.663928] Key type ._llcrypt unregistered [ 1786.203990] Key type ._llcrypt registered [ 1786.207235] Key type .llcrypt registered [ 1786.286461] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 1787.365606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1787.381497] alg: No test for adler32 (adler32-zlib) [ 1788.474639] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 1788.745800] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 1790.423207] Key type lgssc registered [ 1791.460415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1797.172996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1797.185454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.380861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1798.399725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1798.539212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.121696] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1811.007700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.169881] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.209896] Lustre: 57612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1811.529851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1816.549229] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1824.011564] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1830.038964] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1836.855027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1837.110225] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1837.384186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.077525] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 1847.448946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1850.054238] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1856.253246] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2619h20m27s [ 1857.697903] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1865.357727] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1868.069544] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1868.122068] LustreError: 57619:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1888.551458] Lustre: lustre-MDT0000: Client 702ccb10-ca4a-46c4-8464-4801ce6d7335 (at 192.168.204.22@tcp) reconnecting [ 1908.143614] LustreError: 57619:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1910.351193] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1912.793193] Lustre: DEBUG MARKER: done [ 1918.946191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.956818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.987066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.465157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.468037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.492292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.492300] Lustre: Skipped 1 previous similar message [ 1920.931157] Lustre: server umount lustre-OST0000 complete [ 1929.188519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.195185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.207805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.622288] Lustre: server umount lustre-MDT0000 complete [ 1935.876765] LustreError: 59474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786507 with bad export cookie 15423310171110512300 [ 1935.882154] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1935.897965] LustreError: 59474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1936.491854] Lustre: server umount lustre-MDT0001 complete [ 1948.480721] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 1951.578551] Key type lgssc unregistered [ 1951.994354] LNet: 61949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.009338] LNetError: 61949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.044787] LNet: Removed LNI 192.168.204.122@tcp [ 1953.004948] Key type .llcrypt unregistered [ 1953.013817] Key type ._llcrypt unregistered [ 2001.299030] Key type ._llcrypt registered [ 2001.301732] Key type .llcrypt registered [ 2001.428729] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2002.625083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.813562] alg: No test for adler32 (adler32-zlib) [ 2004.210501] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 2004.603023] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2006.311222] Key type lgssc registered [ 2007.983823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2011.826517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2011.850060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.216811] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2013.271854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.455242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.519241] Lustre: Failing over lustre-MDT0000 [ 2014.861974] Lustre: server umount lustre-MDT0000 complete [ 2022.625899] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:23:13 (1787786593) [ 2029.854155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.230215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.561843] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2044.702362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2044.856258] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2044.926579] Lustre: 64746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2045.286873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2050.937492] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2060.284484] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2067.578518] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2075.900033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2076.112189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2076.336354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.235265] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2086.546524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2092.582129] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2098.438669] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2106.849175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2106.865049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.878576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.492564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2112.506896] Lustre: Skipped 5 previous similar messages [ 2117.604791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.613862] Lustre: Skipped 2 previous similar messages [ 2120.159131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2120.493724] Lustre: server umount lustre-MDT0000 complete [ 2124.640197] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786696 with bad export cookie 17322727082571674335 [ 2124.644453] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786696 with bad export cookie 17322727082571674335 [ 2124.653179] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2124.682557] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2125.109924] Lustre: server umount lustre-MDT0001 complete [ 2129.330552] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2134.148695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2142.519130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.968603] Lustre: 63674:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786699/real 1787786699] req@000000008b4a6aea x1874630096759424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.015833] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.031688] Lustre: Skipped 2 previous similar messages [ 2145.247159] Lustre: 63672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786701/real 1787786701] req@00000000017702b2 x1874630096759552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2148.383507] Lustre: 63672:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786704/real 1787786704] req@00000000bd2bde15 x1874630096759680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786720 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.424113] LustreError: 66599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.445586] LustreError: 66599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2149.508559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2149.602627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2150.496124] Lustre: 63673:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786706/real 1787786707] req@00000000ac1912bc x1874630096759936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2152.548314] LustreError: 66600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.554652] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2154.256790] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2154.659495] LustreError: 68521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.670173] LustreError: 68521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2156.705088] Lustre: 63673:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786712/real 1787786712] req@00000000fe05eaaf x1874630096760064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.105170] LustreError: 66599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.124286] LustreError: 66599:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2163.179773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.530195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.928735] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2168.815934] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2174.309560] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.448462] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.281735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.285710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.295337] Lustre: Skipped 1 previous similar message [ 2192.254341] Lustre: server umount lustre-MDT0000 complete [ 2194.407311] LustreError: 69260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.430768] LustreError: 69260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2195.977787] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786767 with bad export cookie 17322727082571676610 [ 2195.983996] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.999843] LustreError: 64741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2196.444398] Lustre: server umount lustre-MDT0001 complete [ 2202.841783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2212.096468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.906947] Lustre: 63674:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787786771/real 1787786771] req@00000000aabf6788 x1874630096786560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787786787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.930981] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.947726] Lustre: Skipped 1 previous similar message [ 2221.089141] LustreError: 63670:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000298ffad1 x1874630096787584/t0(0) o250->MGC192.168.204.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2221.615251] LustreError: 66599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.686922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.735402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2227.126762] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2237.274594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2242.618130] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2243.056692] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2250.180778] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2255.416088] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.793697] Lustre: server umount lustre-MDT0000 complete [ 2263.524865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.542431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.565136] LustreError: 66600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2263.585953] LustreError: 66600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2265.840025] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787786837 with bad export cookie 17322727082571677989 [ 2265.845889] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.853358] LustreError: 64740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2266.189295] Lustre: server umount lustre-MDT0001 complete [ 2271.442161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2274.759638] Lustre: DEBUG MARKER: files compared the same [ 2287.456836] Lustre: server umount lustre-OST0000 complete [ 2306.184945] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2309.364171] Key type lgssc unregistered [ 2309.649586] LNet: 74488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.658599] LNetError: 74488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.677569] LNet: Removed LNI 192.168.204.122@tcp [ 2310.764497] Key type .llcrypt unregistered [ 2310.767429] Key type ._llcrypt unregistered [ 2344.755520] Key type ._llcrypt registered [ 2344.757218] Key type .llcrypt registered [ 2344.954204] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2346.206613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.300307] alg: No test for adler32 (adler32-zlib) [ 2347.418907] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 2347.660958] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2349.383317] Key type lgssc registered [ 2350.882259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2359.780214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2359.807201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2361.200402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2365.492830] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug +malloc all [ 2371.552815] LustreError: 75905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2371.575474] LustreError: 75905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2374.319150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.582467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.058324] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug +malloc all [ 2386.291966] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.841673] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2398.393363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.715627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.082514] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug +malloc all [ 2406.904719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2413.421115] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2425.311738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2425.320921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.333767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.359774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2427.364670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.372833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.495713] Lustre: server umount lustre-OST0000 complete [ 2436.064068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2436.074216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.096436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2436.111109] Lustre: Skipped 1 previous similar message [ 2442.720064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.726021] Lustre: Skipped 2 previous similar messages [ 2447.841109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2447.848992] Lustre: Skipped 1 previous similar message [ 2449.375139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2449.682370] Lustre: server umount lustre-MDT0000 complete [ 2452.976417] LustreError: 77388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.007714] LustreError: 77388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2454.249080] LustreError: 76640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787026 with bad export cookie 10976857061365769926 [ 2454.249637] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2454.257697] LustreError: 76640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2454.914860] Lustre: server umount lustre-MDT0001 complete [ 2466.957218] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2470.028933] Key type lgssc unregistered [ 2470.343292] LNet: 79443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.348815] LNetError: 79443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2470.369576] LNet: Removed LNI 192.168.204.122@tcp [ 2471.384228] Key type .llcrypt unregistered [ 2471.386516] Key type ._llcrypt unregistered [ 2494.990731] Key type ._llcrypt registered [ 2494.992610] Key type .llcrypt registered [ 2495.095278] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2496.017273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2496.171273] alg: No test for adler32 (adler32-zlib) [ 2497.331532] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 2497.672312] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2499.359236] Key type lgssc registered [ 2500.422875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2507.835649] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2507.851968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.424277] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2528.543862] LustreError: 80836:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.122@tcp: failed processing log, type 4: rc = -110 [ 2555.167360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.096994] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2570.360479] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2577.853108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.015197] Lustre: *** cfs_fail_loc=706, val=0*** [ 2578.375473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.517884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2582.079896] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2583.524301] LustreError: 80862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.550556] LustreError: 80862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2588.647341] LustreError: 80861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.686329] LustreError: 81922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.709537] LustreError: 81922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2590.100326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.438495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2595.276733] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2603.556317] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2610.077036] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2614.241330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.254241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.287710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.297717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2616.298207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.336068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.874579] Lustre: server umount lustre-OST0000 complete [ 2626.533459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.533672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.547483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.547491] Lustre: Skipped 1 previous similar message [ 2626.568203] Lustre: Skipped 1 previous similar message [ 2629.896675] Lustre: server umount lustre-MDT0000 complete [ 2631.655404] LustreError: 82655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.678763] LustreError: 82655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2634.331565] LustreError: 80844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787206 with bad export cookie 17947577925767933496 [ 2634.335143] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.351976] LustreError: 80844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2634.960989] Lustre: server umount lustre-MDT0001 complete [ 2645.760127] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 2648.298220] Key type lgssc unregistered [ 2648.554521] LNet: 84455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2648.560028] LNetError: 84455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2648.587651] LNet: Removed LNI 192.168.204.122@tcp [ 2649.259791] Key type .llcrypt unregistered [ 2649.261921] Key type ._llcrypt unregistered [ 2672.450588] Key type ._llcrypt registered [ 2672.454598] Key type .llcrypt registered [ 2672.559875] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2673.716865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2673.735149] alg: No test for adler32 (adler32-zlib) [ 2674.811138] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 2675.107364] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 2676.880410] Key type lgssc registered [ 2678.208463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.001799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2687.033110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2692.870742] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2700.059717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.694594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2704.996729] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2705.891446] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.923766] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2711.030875] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.064380] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2715.619387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2716.099659] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2716.135116] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2716.158492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.252326] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.283788] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2723.360184] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2726.371738] LustreError: 86562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.405470] LustreError: 86562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2732.593187] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.604126] Lustre: MGS: Client 28d5be6a-8b0e-46fe-83b9-4a804e9cf75b (at 0@lo) reconnecting [ 2732.636763] LustreError: 86572:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2732.871287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.974179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2737.447429] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2753.513392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.518426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2753.540268] Lustre: Skipped 1 previous similar message [ 2753.540642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2753.544210] Lustre: Skipped 1 previous similar message [ 2757.218558] Lustre: server umount lustre-OST0000 complete [ 2763.747927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.779770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.784641] Lustre: Skipped 1 previous similar message [ 2766.505227] Lustre: server umount lustre-MDT0000 complete [ 2768.864930] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.881532] LustreError: 86561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2770.191193] Lustre: server umount lustre-MDT0001 complete [ 2773.684215] Lustre: server umount MGS complete [ 2784.379948] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:35:56 (1787787356) [ 2813.024505] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 2822.667867] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 2862.486281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.025532] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2877.446515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.719226] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2877.781103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2877.899808] Lustre: lustre-MDT0001: new disk, initializing [ 2878.006422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.055243] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2878.066478] Lustre: 93730:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2878.085367] LustreError: 93730:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2882.111172] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2893.163356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2893.426780] Lustre: lustre-OST0000: new disk, initializing [ 2893.431615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2893.438983] Lustre: 94495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2893.508914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.261812] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2900.576062] Lustre: 94792:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a1f9974c x1874630800351488/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787787488 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2909.776638] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2916.370149] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2916.382911] Lustre: MGS: Client a7715dca-f683-4896-9793-f55f1fa42f1b (at 0@lo) reconnecting [ 2916.440669] Lustre: 92996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2916.485687] LustreError: 93716:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2916.498590] LustreError: 93716:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2916.520186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2916.580536] Lustre: lustre-MDT0000: new disk, initializing [ 2916.747192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.832046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2916.877820] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2917.920587] Lustre: 95565:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000779c7f1d x1874630800358400/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787787505 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2921.505586] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 2923.039842] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2927.019600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2927.026641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2927.110435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2951.648555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2951.653170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2951.664400] Lustre: Skipped 2 previous similar messages [ 2951.676258] Lustre: Skipped 1 previous similar message [ 2956.575161] Lustre: server umount lustre-OST0000 complete [ 2961.888957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.889257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2961.901310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.582278] Lustre: server umount lustre-MDT0000 complete [ 2967.009735] LustreError: 93700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2967.036583] LustreError: 93700:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2976.379127] Lustre: server umount lustre-MDT0001 complete [ 2980.143498] Lustre: server umount MGS complete [ 2991.905260] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:39:23 (1787787563) [ 3012.306688] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3015.591211] Key type lgssc unregistered [ 3016.004288] LNet: 98070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.015323] LNetError: 98070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.032385] LNet: Removed LNI 192.168.204.122@tcp [ 3016.811757] Key type .llcrypt unregistered [ 3016.813428] Key type ._llcrypt unregistered [ 3033.608232] Key type ._llcrypt registered [ 3033.609766] Key type .llcrypt registered [ 3033.762209] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3035.496183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3035.549539] alg: No test for adler32 (adler32-zlib) [ 3036.621370] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3036.882626] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3038.519178] Key type lgssc registered [ 3039.656247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3048.108555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3048.135934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.695246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.767960] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3059.180604] LustreError: 99295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.203226] LustreError: 99295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3064.293623] LustreError: 99293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3064.509262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3064.902788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3070.067113] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3077.514933] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3083.972672] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3091.507926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3091.803093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3094.903106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3099.331198] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3107.344245] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3119.589783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3119.591475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.620647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3119.624505] Lustre: Skipped 1 previous similar message [ 3122.677535] Lustre: server umount lustre-OST0000 complete [ 3129.825139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.827447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.843300] Lustre: Skipped 1 previous similar message [ 3129.865905] Lustre: Skipped 2 previous similar messages [ 3133.248664] Lustre: server umount lustre-MDT0000 complete [ 3134.952368] LustreError: 99293:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.964374] LustreError: 99293:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3136.859659] LustreError: 101139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787708 with bad export cookie 10373832329772059068 [ 3136.861920] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3136.867691] LustreError: 101139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3137.198937] Lustre: server umount lustre-MDT0001 complete [ 3148.839439] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3151.405539] Key type lgssc unregistered [ 3151.688058] LNet: 102831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3151.694837] LNetError: 102831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.746251] LNet: Removed LNI 192.168.204.122@tcp [ 3153.581618] Key type .llcrypt unregistered [ 3153.583589] Key type ._llcrypt unregistered [ 3171.162412] Key type ._llcrypt registered [ 3171.164040] Key type .llcrypt registered [ 3171.278503] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3172.414817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3172.512376] alg: No test for adler32 (adler32-zlib) [ 3173.793661] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3174.344859] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3176.183186] Key type lgssc registered [ 3177.297850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3183.613110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3183.615801] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3183.627927] Lustre: Skipped 1 previous similar message [ 3183.627951] LustreError: 103917:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3183.660557] LustreError: 103918:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3183.667170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.669152] LustreError: 103918:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3183.669175] LustreError: 103918:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3185.244521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3193.360478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.729758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3198.929290] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3205.304938] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing lsmod [ 3208.407726] LustreError: 105124:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3208.443974] LustreError: 105127:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3208.449241] LustreError: 105124:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4963 [ 3208.454697] LustreError: 105124:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3208.462310] LustreError: 105124:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3208.475198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.937157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3211.003848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3214.304740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.315333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.340038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.353431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3216.357246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3216.368727] Lustre: Skipped 1 previous similar message [ 3216.393857] Lustre: Skipped 1 previous similar message [ 3218.465871] LustreError: 103951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.490893] LustreError: 103951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3220.408520] Lustre: server umount lustre-MDT0000 complete [ 3223.527163] LustreError: 103956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.545330] LustreError: 103956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3224.210310] LustreError: 103932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787796 with bad export cookie 17802796244262870469 [ 3224.218766] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.223029] LustreError: 103932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.631624] LustreError: 104387:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cff1dbe0 x1874631323572352/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3224.799400] Lustre: server umount lustre-MDT0001 complete [ 3239.144794] Lustre: server umount lustre-OST0000 complete [ 3248.243668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.722344] LustreError: 106189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.803174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.600337] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3254.245717] LustreError: 106190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3258.354627] LustreError: 106189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.734188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.221050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.094381] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3276.961267] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3284.366719] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3292.568737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.060625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.107268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3300.242670] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3310.450172] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3318.760037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.763193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3318.766325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3318.769355] Lustre: Skipped 1 previous similar message [ 3318.773700] Lustre: Skipped 1 previous similar message [ 3323.880490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.888542] Lustre: Skipped 1 previous similar message [ 3324.303613] Lustre: server umount lustre-OST0000 complete [ 3328.995705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.996654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3329.009196] Lustre: Skipped 1 previous similar message [ 3329.023849] Lustre: Skipped 1 previous similar message [ 3334.602960] Lustre: server umount lustre-MDT0000 complete [ 3339.236586] LustreError: 106190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.259594] LustreError: 106190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3339.277367] LustreError: 106169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787787911 with bad export cookie 17802796244262871400 [ 3339.294366] LustreError: 106169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3339.294942] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.811954] Lustre: server umount lustre-MDT0001 complete [ 3352.122979] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3354.742593] Key type lgssc unregistered [ 3355.086610] LNet: 109736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3355.096802] LNetError: 109736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3355.123908] LNet: Removed LNI 192.168.204.122@tcp [ 3356.253423] Key type .llcrypt unregistered [ 3356.256196] Key type ._llcrypt unregistered [ 3381.756979] Key type ._llcrypt registered [ 3381.759663] Key type .llcrypt registered [ 3381.881199] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3383.293605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3383.425443] alg: No test for adler32 (adler32-zlib) [ 3384.516844] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3384.770498] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3386.527315] Key type lgssc registered [ 3387.929449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3396.245547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3396.332525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3397.960962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.318358] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3407.339702] LustreError: 111155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.361458] LustreError: 111155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3412.451609] LustreError: 111156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.592936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3414.022442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3418.981894] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3427.875654] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3433.602059] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3440.860314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3441.334648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3442.394465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3448.435092] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3457.173712] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3463.006586] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3467.520976] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3473.385130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3473.394383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.419191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3474.401394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3474.429295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3479.120054] Lustre: server umount lustre-OST0000 complete [ 3484.648470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.671126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3484.679368] Lustre: Skipped 1 previous similar message [ 3484.694472] Lustre: Skipped 2 previous similar messages [ 3489.761379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.776096] Lustre: Skipped 1 previous similar message [ 3490.806623] Lustre: server umount lustre-MDT0000 complete [ 3494.325036] LustreError: 111137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788066 with bad export cookie 1182630328109083791 [ 3494.338530] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3494.345158] LustreError: 111137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3494.669309] Lustre: server umount lustre-MDT0001 complete [ 3506.177584] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3509.358750] Key type lgssc unregistered [ 3509.652641] LNet: 114797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3509.661709] LNetError: 114797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3509.685946] LNet: Removed LNI 192.168.204.122@tcp [ 3510.776295] Key type .llcrypt unregistered [ 3510.778132] Key type ._llcrypt unregistered [ 3529.140069] Key type ._llcrypt registered [ 3529.150617] Key type .llcrypt registered [ 3529.357986] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3530.867965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3531.006204] alg: No test for adler32 (adler32-zlib) [ 3532.402981] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3532.745030] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3534.447487] Key type lgssc registered [ 3536.174384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3545.673765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.694470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.308354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.728408] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3557.869076] LustreError: 116022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.901854] LustreError: 116022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3562.977244] LustreError: 116021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3563.888378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3564.214190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.539102] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3579.708285] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3587.177241] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3596.270603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3596.699402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.224908] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3611.997648] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3617.240261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3627.494677] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3627.496537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3627.505495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3627.511351] Lustre: Skipped 1 previous similar message [ 3632.614119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.628386] Lustre: Skipped 1 previous similar message [ 3633.425317] Lustre: server umount lustre-OST0000 complete [ 3637.728969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.730542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3637.754602] Lustre: Skipped 1 previous similar message [ 3637.775410] Lustre: Skipped 2 previous similar messages [ 3642.851880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3642.856099] Lustre: Skipped 1 previous similar message [ 3643.894908] Lustre: server umount lustre-MDT0000 complete [ 3647.972561] LustreError: 116021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.990229] LustreError: 116021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3648.323236] LustreError: 116002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788220 with bad export cookie 11702395050934012114 [ 3648.331755] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3648.332595] LustreError: 116002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3648.591652] Lustre: server umount lustre-MDT0001 complete [ 3662.756685] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3665.999932] Key type lgssc unregistered [ 3666.399745] LNet: 119580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3666.405521] LNetError: 119580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.420264] LNet: Removed LNI 192.168.204.122@tcp [ 3667.362542] Key type .llcrypt unregistered [ 3667.365854] Key type ._llcrypt unregistered [ 3691.009685] Key type ._llcrypt registered [ 3691.013362] Key type .llcrypt registered [ 3691.123948] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3692.019263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3692.182084] alg: No test for adler32 (adler32-zlib) [ 3693.239375] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3693.466725] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3695.143592] Key type lgssc registered [ 3696.541508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3705.517658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3705.545401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.190243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3712.767421] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3717.602377] LustreError: 120999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.630804] LustreError: 120999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3721.931443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.296495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3726.621932] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3734.381726] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3741.968881] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3749.220832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3749.617945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.697678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3756.413314] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3765.947766] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3774.690296] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3779.508667] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3779.520982] Lustre: Skipped 2 previous similar messages [ 3780.854660] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3780.884710] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3785.196557] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3785.214979] Lustre: Skipped 2 previous similar messages [ 3787.832309] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3787.844425] Lustre: Skipped 1 previous similar message [ 3794.075262] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.22@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3794.081522] Lustre: Skipped 2 previous similar messages [ 3795.982871] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.22@tcp 192.168.0.[2,10]@tcp [ 3802.278214] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.22/24@tcp in log lustre-MDT0000 [ 3802.289766] Lustre: Skipped 2 previous similar messages [ 3805.375559] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.22/24@tcp [ 3805.383463] Lustre: Skipped 1 previous similar message [ 3813.345633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3813.347687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3813.356925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.367827] Lustre: Skipped 1 previous similar message [ 3818.471095] LustreError: 122998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.508605] LustreError: 122998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3818.670710] Lustre: server umount lustre-OST0000 complete [ 3823.584935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.599683] Lustre: Skipped 1 previous similar message [ 3823.605982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.618633] Lustre: Skipped 1 previous similar message [ 3828.705709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3828.718179] Lustre: Skipped 1 previous similar message [ 3830.048447] Lustre: server umount lustre-MDT0000 complete [ 3833.826620] LustreError: 122238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.844836] LustreError: 122238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3834.582831] LustreError: 120979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788406 with bad export cookie 9405037536970812096 [ 3834.594836] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.595774] LustreError: 120979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3835.009414] Lustre: server umount lustre-MDT0001 complete [ 3846.387219] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 3849.868764] Key type lgssc unregistered [ 3850.265342] LNet: 125706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3850.308547] LNetError: 125706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.377675] LNet: Removed LNI 192.168.204.122@tcp [ 3852.288957] Key type .llcrypt unregistered [ 3852.291912] Key type ._llcrypt unregistered [ 3881.207224] Key type ._llcrypt registered [ 3881.209688] Key type .llcrypt registered [ 3881.292516] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 3882.694166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3882.723931] alg: No test for adler32 (adler32-zlib) [ 3883.834657] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 3884.141665] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 3885.921740] Key type lgssc registered [ 3887.462923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3895.567681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3895.622102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3897.084643] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3897.106466] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.22/24@tcp [ 3897.130773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3902.127286] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3907.558843] LustreError: 127280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3907.576732] LustreError: 127280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3911.099256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3911.551633] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3911.567188] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.22/24@tcp [ 3911.647168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.688931] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3925.716346] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3931.867716] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3939.650559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3940.059811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.163646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3947.402396] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 3955.462298] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3965.927074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3965.937252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.949208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.948937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3966.954218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3966.956780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.993243] Lustre: Skipped 1 previous similar message [ 3971.150713] Lustre: server umount lustre-OST0000 complete [ 3977.184936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.208950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3977.216341] Lustre: Skipped 1 previous similar message [ 3981.055421] Lustre: server umount lustre-MDT0000 complete [ 3982.306290] LustreError: 127279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3982.318710] LustreError: 127279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3984.881806] LustreError: 127261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788556 with bad export cookie 2454797329018350138 [ 3984.889202] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3984.894638] LustreError: 127261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3985.175882] Lustre: server umount lustre-MDT0001 complete [ 3996.724615] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 4000.245247] Key type lgssc unregistered [ 4000.655382] LNet: 130869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4000.664111] LNetError: 130869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4000.698439] LNet: Removed LNI 192.168.204.122@tcp [ 4001.710679] Key type .llcrypt unregistered [ 4001.715918] Key type ._llcrypt unregistered [ 4029.016545] Key type ._llcrypt registered [ 4029.018718] Key type .llcrypt registered [ 4029.153445] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 4044.434488] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 4045.769138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.950872] alg: No test for adler32 (adler32-zlib) [ 4047.108158] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 4047.445971] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 4049.290739] Key type lgssc registered [ 4050.885618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4088.691274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4088.722700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.978463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4090.067480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4090.207407] Lustre: lustre-MDT0000: new disk, initializing [ 4090.343981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4090.375591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4095.034458] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4106.632138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.747374] Lustre: 134271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4106.845212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4106.856601] Lustre: Skipped 1 previous similar message [ 4106.983780] Lustre: lustre-MDT0001: new disk, initializing [ 4107.135279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4107.184078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4107.193265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.413295] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4121.834983] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4128.729909] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4135.931107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4136.153313] Lustre: lustre-OST0000: new disk, initializing [ 4136.159144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4136.171207] Lustre: 136241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4136.274311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4142.715212] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4146.255439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4146.265936] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4146.410031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4151.414362] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4156.326314] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4156.561988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4161.681757] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4162.065671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4167.136397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4167.148199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.155495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4167.167100] Lustre: Skipped 1 previous similar message [ 4167.175250] Lustre: Skipped 1 previous similar message [ 4170.662043] Lustre: server umount lustre-OST0000 complete [ 4177.376893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.394206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.406308] Lustre: Skipped 1 previous similar message [ 4181.406825] Lustre: server umount lustre-MDT0000 complete [ 4185.943940] LustreError: 134261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788757 with bad export cookie 988132272643830850 [ 4185.949076] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.953262] LustreError: 134261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4186.284454] Lustre: server umount lustre-MDT0001 complete [ 4195.537810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.050625] LustreError: 138457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4196.065868] LustreError: 138457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4196.119602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4200.975853] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4201.453401] LustreError: 138458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.577838] LustreError: 138457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.932382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.383468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.127588] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4224.341871] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4230.962798] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4238.043669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.501948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.131000] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4252.653292] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4257.662092] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4257.895910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4262.331627] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4262.521317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4270.102638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.237572] Lustre: lustre-OST0001: new disk, initializing [ 4270.241438] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4270.248252] Lustre: 141830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4270.320298] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4275.946142] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4279.401165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4279.415152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4279.534538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4284.917721] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4285.279987] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4290.993996] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4291.201379] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4296.576690] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99edc986c800.ost_server_uuid 50 [ 4297.999399] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99edc986c800.ost_server_uuid in IDLE state after 0 sec [ 4305.383802] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.385548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4305.390328] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4305.390339] Lustre: Skipped 1 previous similar message [ 4305.395378] Lustre: Skipped 1 previous similar message [ 4305.415167] LustreError: Skipped 2 previous similar messages [ 4310.500037] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4310.506159] Lustre: Skipped 2 previous similar messages [ 4315.621629] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4315.629234] Lustre: Skipped 1 previous similar message [ 4319.202402] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4319.348145] Lustre: server umount lustre-OST0001 complete [ 4320.744647] LustreError: 140281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4320.766803] LustreError: 140281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4322.796920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4322.804674] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.813238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4323.823936] Lustre: Skipped 1 previous similar message [ 4325.867030] LustreError: 142332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.883327] LustreError: 142332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4328.615574] Lustre: server umount lustre-OST0000 complete [ 4333.539994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4333.551074] LustreError: Skipped 1 previous similar message [ 4333.557446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.569580] Lustre: Skipped 1 previous similar message [ 4341.216281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4341.225038] Lustre: Skipped 6 previous similar messages [ 4346.335576] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4348.484495] Lustre: server umount lustre-MDT0000 complete [ 4350.440744] LustreError: 139660:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.465676] LustreError: 139660:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4351.564185] LustreError: 139193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787788923 with bad export cookie 988132272643831732 [ 4351.581032] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4351.581940] LustreError: 139193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4351.870427] Lustre: server umount lustre-MDT0001 complete [ 4368.115869] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 4370.748349] Key type lgssc unregistered [ 4370.993257] LNet: 144852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4370.999399] LNetError: 144852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.027534] LNet: Removed LNI 192.168.204.122@tcp [ 4371.771848] Key type .llcrypt unregistered [ 4371.776110] Key type ._llcrypt unregistered [ 4408.770331] Key type ._llcrypt registered [ 4408.775804] Key type .llcrypt registered [ 4408.886571] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 4420.872851] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 4421.696410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4421.707835] alg: No test for adler32 (adler32-zlib) [ 4422.751755] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 4422.944395] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 4424.599231] Key type lgssc registered [ 4425.425838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4452.936643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4452.977703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.212139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4454.251481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4454.335222] Lustre: lustre-MDT0000: new disk, initializing [ 4454.427218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.454090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4457.848190] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4466.639811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.763902] Lustre: 148982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4466.815527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4466.818643] Lustre: Skipped 1 previous similar message [ 4466.869900] Lustre: lustre-MDT0001: new disk, initializing [ 4466.940226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4466.975894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4466.983864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4470.536895] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4478.054260] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4482.818651] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4487.928396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.155975] Lustre: lustre-OST0000: new disk, initializing [ 4488.162626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4488.169202] Lustre: 150954:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4488.242176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.223322] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4494.896515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4494.905924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4494.975956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4500.464073] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4503.972036] Lustre: Failing over lustre-OST0000 [ 4504.131178] Lustre: server umount lustre-OST0000 complete [ 4521.753205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4521.951128] Lustre: 147210:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789077/real 1787789077] req@000000004592df04 x1874632632204416/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787789093 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4521.951323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.973193] Lustre: 147210:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4522.010480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4522.027872] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4522.275567] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4523.039814] Lustre: 147207:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789079/real 1787789079] req@000000007cccc50c x1874632632204544/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787789095 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4523.063717] Lustre: 147207:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4526.176465] Lustre: 147207:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789082/real 1787789082] req@0000000070fe960a x1874632632205568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787789098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4527.098018] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4527.100531] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4527.222313] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4530.673029] Lustre: Failing over lustre-MDT0000 [ 4530.886674] Lustre: server umount lustre-MDT0000 complete [ 4531.680373] Lustre: 147207:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789087/real 1787789087] req@000000000d32ecc7 x1874632632206720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787789103 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4531.721668] Lustre: 147207:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4532.198737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.216036] Lustre: Skipped 3 previous similar messages [ 4536.287317] Lustre: 147210:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789092/real 1787789092] req@00000000a52ae975 x1874632632207872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787789108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4537.321862] LustreError: 148995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4537.335709] LustreError: 148995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4542.439611] LustreError: 150149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.455869] LustreError: 150149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4547.040339] Lustre: 148987:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787789103/real 1787789103] req@000000002b8652f0 x1874632632215552/t0(0) o101->MGC192.168.204.122@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1787789119 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4547.059978] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4547.068557] Lustre: 148987:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.122@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4547.073485] LustreError: 148992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4547.092281] LustreError: 148992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4548.297816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4552.160477] LustreError: 150149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.179955] LustreError: 150149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4557.280413] LustreError: 148995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4557.290981] LustreError: 148995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4557.470936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.538355] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4558.072483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4561.381960] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4562.928192] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4562.939950] Lustre: Skipped 1 previous similar message [ 4562.965639] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4563.011119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4568.035208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4568.039502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.048977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4568.055232] Lustre: Skipped 1 previous similar message [ 4570.006895] Lustre: server umount lustre-OST0000 complete [ 4573.160590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.162719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4573.175070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4573.191669] Lustre: Skipped 2 previous similar messages [ 4578.274829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4578.279473] Lustre: Skipped 1 previous similar message [ 4579.347689] Lustre: server umount lustre-MDT0000 complete [ 4581.888581] LustreError: 153861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789153 with bad export cookie 17818148768484547135 [ 4581.895069] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.897644] LustreError: 153861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4582.186962] Lustre: server umount lustre-MDT0001 complete [ 4589.482066] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 4591.767846] Key type lgssc unregistered [ 4592.019602] LNet: 154615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4592.027995] LNetError: 154615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4592.040058] LNet: Removed LNI 192.168.204.122@tcp [ 4592.853861] Key type .llcrypt unregistered [ 4592.856362] Key type ._llcrypt unregistered [ 4624.811815] Key type ._llcrypt registered [ 4624.816534] Key type .llcrypt registered [ 4624.943219] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 4625.752122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4625.776416] alg: No test for adler32 (adler32-zlib) [ 4626.839662] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 4627.220037] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 4628.943448] Key type lgssc registered [ 4630.398515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4639.123950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4639.144241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.606344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.123890] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4650.987093] LustreError: 156135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.010664] LustreError: 156135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4654.309804] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4656.101443] LustreError: 156134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.769897] LustreError: 156163:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4657.785789] LustreError: 156163:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4657.792576] LustreError: 156163:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4658.178379] Lustre: server umount lustre-MDT0000 complete [ 4671.578870] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 4679.478495] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 4709.983738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4710.153682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4710.184637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4710.261470] Lustre: lustre-MDT0000: new disk, initializing [ 4710.347573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4710.363387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4714.598642] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4723.590278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4723.701938] Lustre: 159483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4723.808552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4723.813777] Lustre: Skipped 1 previous similar message [ 4723.885443] Lustre: lustre-MDT0001: new disk, initializing [ 4723.987091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4724.034601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4724.043899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4728.072830] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4736.358401] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4741.306556] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4747.018098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.282558] Lustre: lustre-OST0000: new disk, initializing [ 4747.293426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4747.302225] Lustre: 161450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4747.372065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.876469] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 4753.976242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4753.985736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4754.121575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4760.067984] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4764.088993] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4765.350320] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:08:57 (1787789337) [ 4995.042268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4995.055881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4995.059651] Lustre: Skipped 1 previous similar message [ 4995.075694] Lustre: Skipped 2 previous similar messages [ 4997.362452] Lustre: server umount lustre-MDT0000 complete [ 5000.160425] LustreError: 159493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5000.181705] LustreError: 159493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5000.396973] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789572 with bad export cookie 13063565871681899451 [ 5000.397950] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5000.412613] LustreError: 159473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5005.280052] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.280052] LustreError: 163204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5005.308422] Lustre: Skipped 1 previous similar message [ 5005.314453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5006.706889] Lustre: server umount lustre-MDT0001 complete [ 5015.820062] Lustre: server umount lustre-OST0000 complete [ 5023.075885] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 5030.562172] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5059.156322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.334517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5059.367583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5059.427560] Lustre: lustre-MDT0000: new disk, initializing [ 5059.477505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5059.497479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5062.843965] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5070.957564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.043220] Lustre: 166106:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5071.085325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5071.089122] Lustre: Skipped 1 previous similar message [ 5071.143698] Lustre: lustre-MDT0001: new disk, initializing [ 5071.208081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5071.235586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5071.242993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5074.278115] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5081.736554] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5086.080712] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5090.849060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5091.043743] Lustre: lustre-OST0000: new disk, initializing [ 5091.047483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5091.053946] Lustre: 168072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5091.116303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.946371] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5099.572719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5099.580926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5099.687921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5103.152697] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5106.031808] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5106.202574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5109.513398] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5109.650834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5114.848610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.856396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.865348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5115.360561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5117.247952] Lustre: server umount lustre-OST0000 complete [ 5120.484281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.487538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5120.495103] Lustre: Skipped 2 previous similar messages [ 5120.505797] Lustre: Skipped 2 previous similar messages [ 5125.602028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5125.607723] Lustre: Skipped 2 previous similar messages [ 5134.304762] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5134.429263] Lustre: server umount lustre-MDT0000 complete [ 5135.842820] LustreError: 167493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5135.860294] LustreError: 167493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5136.938588] LustreError: 166902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789708 with bad export cookie 13063565871682058281 [ 5136.946218] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.954073] LustreError: 166902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5137.178764] Lustre: server umount lustre-MDT0001 complete [ 5143.080474] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:15:15 (1787789715) [ 5154.930929] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 5161.840522] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5190.860609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.104690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5191.134298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5191.222229] Lustre: lustre-MDT0000: new disk, initializing [ 5191.278122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.301891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5195.608828] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5205.235091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.348816] Lustre: 172828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5205.540960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5209.263862] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5221.580592] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5227.875762] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5234.312924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5234.544716] Lustre: lustre-OST0000: new disk, initializing [ 5234.547263] Lustre: Skipped 1 previous similar message [ 5234.550557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5234.554295] Lustre: Skipped 2 previous similar messages [ 5234.560942] Lustre: 174805:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5239.529968] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5241.899227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5241.904901] Lustre: Skipped 1 previous similar message [ 5241.907845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5241.978192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5248.073980] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5255.379941] Lustre: server umount lustre-OST0000 complete [ 5261.798764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5261.816712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5261.829726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5261.834601] Lustre: Skipped 2 previous similar messages [ 5268.540752] Lustre: server umount lustre-MDT0000 complete [ 5272.288778] LustreError: 172819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789844 with bad export cookie 13063565871682059156 [ 5272.304216] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.312146] LustreError: 172819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.924304] Lustre: server umount lustre-MDT0001 complete [ 5290.151853] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:17:42 (1787789862) [ 5303.650640] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 5311.336430] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5339.188924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5339.374363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5339.395793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5339.451595] Lustre: lustre-MDT0000: new disk, initializing [ 5339.510373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.514412] Lustre: Skipped 2 previous similar messages [ 5339.532577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5343.043306] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5352.100368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.230879] Lustre: 179123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5352.437628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5356.361459] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5364.699481] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5369.412861] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5375.170181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5375.368944] Lustre: 181098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5379.872467] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5382.778078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5386.448525] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5392.867585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5392.868103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5392.879569] Lustre: Skipped 1 previous similar message [ 5392.881686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5392.898226] Lustre: Skipped 5 previous similar messages [ 5397.565980] Lustre: server umount lustre-OST0000 complete [ 5406.499391] Lustre: server umount lustre-MDT0000 complete [ 5408.226938] LustreError: 179135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.242019] LustreError: 179135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5409.083786] LustreError: 179116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787789981 with bad export cookie 13063565871682060248 [ 5409.088636] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5409.096409] LustreError: 179116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5409.297404] Lustre: server umount lustre-MDT0001 complete [ 5417.489267] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 5419.806871] Key type lgssc unregistered [ 5420.015701] LNet: 182957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5420.021126] LNetError: 182957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5420.031245] LNet: Removed LNI 192.168.204.122@tcp [ 5420.798959] Key type .llcrypt unregistered [ 5420.803299] Key type ._llcrypt unregistered [ 5442.553160] Key type ._llcrypt registered [ 5442.557243] Key type .llcrypt registered [ 5442.694617] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5443.706874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5443.825552] alg: No test for adler32 (adler32-zlib) [ 5444.820820] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 5445.024544] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 5446.679229] Key type lgssc registered [ 5447.636207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5453.853510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5453.870996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.176529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5458.135234] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5463.579934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.846815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5466.589461] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5471.961250] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5475.916049] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5481.066476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5481.339738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5486.207969] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5492.350241] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5500.793357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5520.353371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5520.360471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.390969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.384047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5521.384805] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5521.390382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.393023] Lustre: Skipped 1 previous similar message [ 5525.611843] Lustre: server umount lustre-OST0000 complete [ 5530.592550] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5530.602967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.618823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5534.907636] Lustre: server umount lustre-MDT0000 complete [ 5536.743507] LustreError: 184366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5536.766312] LustreError: 184366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5537.497650] LustreError: 184346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790109 with bad export cookie 3307481662056157532 [ 5537.508857] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5537.515353] LustreError: 184346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5537.799787] Lustre: server umount lustre-MDT0001 complete [ 5546.703554] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 5549.067073] Key type lgssc unregistered [ 5549.304321] LNet: 187924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5549.311272] LNetError: 187924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5549.323816] LNet: Removed LNI 192.168.204.122@tcp [ 5549.980446] Key type .llcrypt unregistered [ 5549.982705] Key type ._llcrypt unregistered [ 5568.310377] Key type ._llcrypt registered [ 5568.313820] Key type .llcrypt registered [ 5568.387711] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5569.105495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5569.117026] alg: No test for adler32 (adler32-zlib) [ 5570.116195] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 5570.330989] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 5571.976059] Key type lgssc registered [ 5573.024387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5579.578357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5579.601354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.981463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.840178] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5590.500462] LustreError: 189321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.516342] LustreError: 189321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5592.311214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.688388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5596.427074] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5602.646729] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5606.984849] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5612.454137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.738775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5617.366891] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5623.849346] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5625.067982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5633.505119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5633.510928] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5633.521591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5634.844985] Lustre: lustre-OST0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 5635.569393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5635.572650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.519249] Lustre: server umount lustre-OST0000 complete [ 5645.606762] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99edd2106800.ost_server_uuid 50 [ 5647.046910] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5660.455742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.459934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.471630] Lustre: Skipped 1 previous similar message [ 5660.493381] Lustre: Skipped 3 previous similar messages [ 5665.674662] Lustre: server umount lustre-MDT0000 complete [ 5665.764527] LustreError: 189320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5665.790625] LustreError: 189320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5668.664822] LustreError: 189301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790240 with bad export cookie 1511090611415748015 [ 5668.669485] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5668.674784] LustreError: 189301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5668.956463] Lustre: server umount lustre-MDT0001 complete [ 5676.026259] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:24:08 (1787790248) [ 5681.450230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.871899] LustreError: 193236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.985288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5685.249187] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5687.265096] LustreError: 193237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.361841] LustreError: 193236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.750534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.072113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5695.440261] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5701.197995] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5705.527780] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5711.314311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5711.716885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5716.520306] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5722.732543] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5726.061667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5728.469656] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5728.637796] Lustre: lustre-OST0001: new disk, initializing [ 5728.647566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5728.657427] Lustre: 196078:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5728.731363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5733.904996] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5735.496014] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5735.511758] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5735.597716] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5742.010763] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5746.152052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5746.157726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5746.162359] Lustre: Skipped 1 previous similar message [ 5746.173788] Lustre: Skipped 1 previous similar message [ 5750.387922] Lustre: server umount lustre-OST0000 complete [ 5751.267672] LustreError: 195060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5751.279464] LustreError: 195060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5754.869841] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5755.059399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5758.759377] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5758.981773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5768.451328] LustreError: 195060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.477050] LustreError: 195060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5776.865195] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5776.867626] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.875201] LustreError: Skipped 1 previous similar message [ 5776.875946] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5776.875950] Lustre: Skipped 1 previous similar message [ 5776.904017] Lustre: Skipped 1 previous similar message [ 5778.184286] Lustre: server umount lustre-OST0001 complete [ 5787.105119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.113116] Lustre: Skipped 4 previous similar messages [ 5787.356633] Lustre: server umount lustre-MDT0000 complete [ 5790.162633] LustreError: 193218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790362 with bad export cookie 1511090611415749079 [ 5790.168828] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.179544] LustreError: 193218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5790.479387] Lustre: server umount lustre-MDT0001 complete [ 5810.467785] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:26:22 (1787790382) [ 5816.571991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.837530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5816.863704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5817.027196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.189073] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5828.334706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.415989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5828.466663] Lustre: 199515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5832.020042] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5837.688642] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5842.898385] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5849.515736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5849.719757] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5855.190695] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5856.336663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5861.813691] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5867.923225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5868.008713] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5873.142646] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 5875.340320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5879.046191] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5881.826668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5881.826755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.834041] LustreError: Skipped 1 previous similar message [ 5881.843903] Lustre: Skipped 2 previous similar messages [ 5881.847892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5881.862521] Lustre: Skipped 1 previous similar message [ 5887.401701] Lustre: server umount lustre-OST0000 complete [ 5889.791155] LustreError: 202785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.812118] LustreError: 202785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5901.280787] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5901.295546] Lustre: Skipped 2 previous similar messages [ 5907.281794] Lustre: server umount lustre-OST0001 complete [ 5916.642184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.648996] Lustre: Skipped 10 previous similar messages [ 5916.834776] Lustre: server umount lustre-MDT0000 complete [ 5920.180831] LustreError: 199510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790492 with bad export cookie 1511090611415750486 [ 5920.186826] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5920.191788] LustreError: 199510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5920.551744] Lustre: server umount lustre-MDT0001 complete [ 5939.634467] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:28:31 (1787790511) [ 5952.971996] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 5961.865224] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 5992.634326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5992.806902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5992.832774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5992.907405] Lustre: lustre-MDT0000: new disk, initializing [ 5992.979468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5992.984907] Lustre: Skipped 3 previous similar messages [ 5992.996289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5996.475326] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6004.873427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6004.954453] Lustre: 207604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6004.995121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6004.999654] Lustre: Skipped 1 previous similar message [ 6005.072164] Lustre: lustre-MDT0001: new disk, initializing [ 6005.175233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6005.184106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6008.960818] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6017.730792] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6023.168067] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6028.750491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6029.004292] Lustre: lustre-OST0000: new disk, initializing [ 6029.009594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6029.017510] Lustre: 209573:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6034.080737] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6034.999361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6035.013563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6035.138313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6042.396475] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6047.370919] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6047.559696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6052.574512] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6052.869657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6055.922554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.929925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.945471] Lustre: Skipped 2 previous similar messages [ 6055.946721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.961747] LustreError: Skipped 1 previous similar message [ 6061.510077] Lustre: server umount lustre-OST0000 complete [ 6070.975602] Lustre: server umount lustre-MDT0000 complete [ 6071.265165] LustreError: 208716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.281915] LustreError: 208716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6074.121284] LustreError: 207596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790646 with bad export cookie 1511090611415752271 [ 6074.133786] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6074.140393] LustreError: 207596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6074.413887] Lustre: server umount lustre-MDT0001 complete [ 6081.776169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.421473] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6092.398398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.914146] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6104.000428] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6108.919973] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6114.962669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6122.148902] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6129.772459] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6134.243169] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6134.431415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6138.765177] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6138.986961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6142.949521] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6142.952565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6142.972395] Lustre: Skipped 4 previous similar messages [ 6142.990734] LustreError: Skipped 1 previous similar message [ 6146.964220] Lustre: server umount lustre-OST0000 complete [ 6152.602090] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6156.647397] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6184.964472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6185.286996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6185.298611] Lustre: Skipped 5 previous similar messages [ 6190.889857] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6197.530887] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6204.439963] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6208.929440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6208.938889] Lustre: Skipped 7 previous similar messages [ 6210.109354] Lustre: server umount lustre-OST0000 complete [ 6219.365350] Lustre: server umount lustre-MDT0000 complete [ 6222.984392] LustreError: 213633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790794 with bad export cookie 1511090611415753153 [ 6222.996930] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6223.002973] LustreError: 213633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6223.334372] Lustre: server umount lustre-MDT0001 complete [ 6230.751358] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:33:22 (1787790802) [ 6237.101723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.484480] LustreError: 217750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6237.502529] LustreError: 217750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6241.198721] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6249.094258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.353771] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6260.383188] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6265.276665] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6270.762110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6276.938983] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6283.816161] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6287.798697] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6288.046631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6292.225954] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6292.435867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6299.074175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6299.297416] Lustre: lustre-OST0001: new disk, initializing [ 6299.302079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6299.307884] Lustre: 221122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6305.674127] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6307.928239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6307.938124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6308.009767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6314.608422] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6319.172378] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6319.400723] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6323.450209] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6323.655451] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6328.800922] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6328.811716] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.825182] Lustre: Skipped 3 previous similar messages [ 6331.454344] Lustre: server umount lustre-OST0001 complete [ 6337.062391] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6337.251716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6341.287993] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6341.515780] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6369.074648] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6374.387668] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6374.407933] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6374.408639] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6374.422756] Lustre: Skipped 1 previous similar message [ 6374.720348] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6382.304232] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6383.948659] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6389.734690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6389.742184] LustreError: Skipped 1 previous similar message [ 6392.027128] Lustre: server umount lustre-OST0001 complete [ 6414.000485] Lustre: server umount lustre-OST0000 complete [ 6423.525030] Lustre: server umount lustre-MDT0000 complete [ 6427.021458] LustreError: 217732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787790999 with bad export cookie 1511090611415754105 [ 6427.026023] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6427.033134] LustreError: 217732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6433.386278] Lustre: server umount lustre-MDT0001 complete [ 6455.307701] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:37:07 (1787791027) [ 6460.698723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.906592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6460.941704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6461.093246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.100074] Lustre: Skipped 5 previous similar messages [ 6464.935720] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6472.745522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.816456] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6472.838266] Lustre: 226360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6477.173632] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6483.522592] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6488.397693] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6494.873071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.113141] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6501.721156] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6508.748394] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6517.488273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.608328] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6523.642553] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6530.186406] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6531.608518] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6535.163262] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6535.337661] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6539.052821] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6539.281760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6544.229854] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99edc4f7d000.ost_server_uuid 50 [ 6545.762384] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99edc4f7d000.ost_server_uuid in IDLE state after 0 sec [ 6546.639279] Lustre: Permanently deactivating lustre-OST0001 [ 6546.654249] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6549.980889] Lustre: Permanently reactivating lustre-OST0001 [ 6555.114627] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.119239] LustreError: Skipped 1 previous similar message [ 6555.123525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6555.129622] Lustre: Skipped 16 previous similar messages [ 6559.477394] Lustre: server umount lustre-OST0001 complete [ 6560.243941] LustreError: 228205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.261140] LustreError: 228205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 6569.384981] Lustre: server umount lustre-OST0000 complete [ 6587.359134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6587.534225] Lustre: server umount lustre-MDT0000 complete [ 6591.179625] LustreError: 226353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791163 with bad export cookie 1511090611415755393 [ 6591.187576] LustreError: 226353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6591.189348] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6591.458591] Lustre: server umount lustre-MDT0001 complete [ 6612.972735] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 20:39:44 (1787791184) [ 6620.034640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.282059] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6620.316150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.323183] Lustre: Skipped 5 previous similar messages [ 6624.908353] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6632.533460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.636594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6632.665838] Lustre: 233055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6636.270633] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6641.779553] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6646.547601] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6651.660270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6651.824960] Lustre: Permanently deactivating lustre-OST0000 [ 6651.836657] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6656.534666] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6657.586826] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6657.594786] LustreError: 188629:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a3f8d6c5 x1874633836367872/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6662.896570] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.895988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.960965] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6667.965301] Lustre: Skipped 1 previous similar message [ 6672.690323] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6677.096409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6678.241708] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.658378] Lustre: Permanently reactivating lustre-OST0000 [ 6679.674962] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6679.680191] Lustre: Skipped 2 previous similar messages [ 6683.452596] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6683.458248] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6683.466630] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6683.475686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6697.445633] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6697.461421] LustreError: Skipped 2 previous similar messages [ 6697.469934] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.485082] Lustre: Skipped 12 previous similar messages [ 6701.022664] Lustre: server umount lustre-OST0001 complete [ 6710.285483] Lustre: server umount lustre-OST0000 complete [ 6720.016944] Lustre: server umount lustre-MDT0000 complete [ 6723.299840] LustreError: 233048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791295 with bad export cookie 1511090611415757045 [ 6723.309893] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6723.314074] LustreError: 233048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6723.671323] Lustre: server umount lustre-MDT0001 complete [ 6732.412368] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 6734.678666] Key type lgssc unregistered [ 6734.969740] LNet: 237936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6734.974949] LNetError: 237936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.986329] LNet: Removed LNI 192.168.204.122@tcp [ 6735.771414] Key type .llcrypt unregistered [ 6735.773157] Key type ._llcrypt unregistered [ 6755.321688] Key type ._llcrypt registered [ 6755.326283] Key type .llcrypt registered [ 6755.445952] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 6756.147717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6756.259977] alg: No test for adler32 (adler32-zlib) [ 6757.319323] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 6757.512051] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 6759.153804] Key type lgssc registered [ 6760.232138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6769.297789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6769.322807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.715723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.862385] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6780.897903] LustreError: 239381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6780.908268] LustreError: 239381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6783.324753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6783.475280] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6783.514677] Lustre: 239369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6783.527677] Lustre: Permanently deactivating lustre-MDT0001 [ 6783.546668] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6783.767636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6787.813181] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6787.829554] LustreError: 239411:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6787.844314] LustreError: 239411:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6787.859389] LustreError: 239411:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6787.947864] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6794.568174] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6799.854808] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6805.061466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6805.290511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.360673] LustreError: 241222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6808.365560] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6810.306722] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6813.667815] LustreError: 241417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.309766] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6816.741805] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6816.752745] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6816.755411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6821.856692] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6821.898359] LustreError: 241223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6822.860885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6823.063469] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6828.532307] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6828.540364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6828.776773] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6836.193763] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6838.953454] Lustre: Permanently reactivating lustre-MDT0001 [ 6838.967684] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6838.973080] Lustre: Skipped 2 previous similar messages [ 6843.712558] Lustre: lustre-MDT0001: Client 81f49f68-c462-4ecd-a8fd-8b4472bf120c (at 192.168.204.22@tcp) reconnecting [ 6849.498090] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.858337] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6855.265758] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6855.459527] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6859.167134] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6859.333203] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6861.388201] Lustre: Permanently deactivating lustre-MDT0001 [ 6861.400490] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6861.404530] Lustre: Skipped 2 previous similar messages [ 6871.622595] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6878.823589] Lustre: server umount lustre-MDT0000 complete [ 6879.712272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6879.722641] LustreError: 240644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6879.735202] Lustre: Skipped 3 previous similar messages [ 6879.777073] LustreError: 240644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6882.271301] LustreError: 240118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791454 with bad export cookie 9099958237883259294 [ 6882.272320] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.280975] LustreError: 240118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6882.568760] Lustre: server umount lustre-MDT0001 complete [ 6891.944808] Lustre: server umount lustre-OST0000 complete [ 6902.088967] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:44:33 (1787791473) [ 6910.781509] Lustre: server umount lustre-OST0001 complete [ 6915.425920] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 6922.762817] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 6951.998053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6952.158067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6952.164679] Lustre: Skipped 2 previous similar messages [ 6952.194836] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6952.259818] Lustre: lustre-MDT0000: new disk, initializing [ 6952.332638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6952.350252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6955.677160] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6963.465372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.549431] Lustre: 247655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6963.604194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6963.607856] Lustre: Skipped 1 previous similar message [ 6963.661871] Lustre: lustre-MDT0001: new disk, initializing [ 6963.735144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6963.761761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6963.777411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6967.218994] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6975.467256] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.805240] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6984.798109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.044193] Lustre: lustre-OST0000: new disk, initializing [ 6985.047792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6985.053463] Lustre: 249623:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6985.114115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.946985] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 6994.984124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6994.999717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6995.096421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6997.153934] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6998.638036] LustreError: 247660:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7005.467685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7005.619772] Lustre: lustre-OST0001: new disk, initializing [ 7005.625603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7005.633520] Lustre: 250736:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7008.647802] LustreError: 247660:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7008.663392] LustreError: 249266:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7010.384737] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7011.374679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7011.381740] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7011.465801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7017.210596] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7018.331231] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7018.703260] LustreError: 249266:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7021.536940] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7021.542620] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7021.554492] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7022.052765] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7022.061957] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7022.068386] Lustre: Skipped 1 previous similar message [ 7023.889522] Lustre: lustre-OST0001: Not available for connect from 192.168.204.22@tcp (stopping) [ 7026.281952] Lustre: server umount lustre-OST0001 complete [ 7027.176520] LustreError: 249609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.186278] LustreError: 249609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7041.507542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7041.511089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7041.517692] Lustre: Skipped 1 previous similar message [ 7041.521523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7047.566248] Lustre: server umount lustre-OST0000 complete [ 7050.726453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7050.737814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.745669] Lustre: Skipped 1 previous similar message [ 7050.748158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7050.750990] Lustre: Skipped 2 previous similar messages [ 7056.459709] Lustre: server umount lustre-MDT0000 complete [ 7059.219414] LustreError: 248451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791631 with bad export cookie 9099958237883262108 [ 7059.224176] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.233485] LustreError: 248451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7065.561081] Lustre: server umount lustre-MDT0001 complete [ 7073.794514] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7075.794679] Key type lgssc unregistered [ 7076.040065] LNet: 252771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7076.051086] LNetError: 252771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7076.084244] LNet: Removed LNI 192.168.204.122@tcp [ 7076.913377] Key type .llcrypt unregistered [ 7076.915261] Key type ._llcrypt unregistered [ 7108.347121] Key type ._llcrypt registered [ 7108.349845] Key type .llcrypt registered [ 7108.496962] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7109.245218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7109.382701] alg: No test for adler32 (adler32-zlib) [ 7110.446364] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7110.709780] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7112.359379] Key type lgssc registered [ 7113.329753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7119.822978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7119.849320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.104877] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7121.121160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7121.297332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7124.788889] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7131.757872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.828717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7131.849517] Lustre: 254960:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7132.055475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.505611] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7141.131924] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7145.916886] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7151.401885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7151.528414] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7151.736068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7157.070902] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7161.951613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7162.639994] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7213.025134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7213.036714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.050621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7213.539688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7213.543729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.564552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7217.359613] Lustre: server umount lustre-OST0000 complete [ 7220.113612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7229.460321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.692376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7233.750744] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7233.775533] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7233.783409] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7233.786900] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7233.806855] Lustre: Skipped 1 previous similar message [ 7239.364444] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7280.099089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7280.107658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.116130] Lustre: Skipped 1 previous similar message [ 7280.120668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7282.009749] Lustre: server umount lustre-OST0000 complete [ 7285.220140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.231344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.240634] Lustre: Skipped 2 previous similar messages [ 7290.337691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.343705] Lustre: Skipped 1 previous similar message [ 7291.465612] Lustre: server umount lustre-MDT0000 complete [ 7294.244780] LustreError: 254951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791866 with bad export cookie 10770676205459051001 [ 7294.246073] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7294.263549] LustreError: 254951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7294.575610] Lustre: server umount lustre-MDT0001 complete [ 7302.854659] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7305.009701] Key type lgssc unregistered [ 7305.233951] LNet: 260382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7305.239962] LNetError: 260382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7305.259415] LNet: Removed LNI 192.168.204.122@tcp [ 7305.969388] Key type .llcrypt unregistered [ 7305.972774] Key type ._llcrypt unregistered [ 7325.576496] Key type ._llcrypt registered [ 7325.578418] Key type .llcrypt registered [ 7325.665178] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7326.537301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7326.549199] alg: No test for adler32 (adler32-zlib) [ 7327.501541] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7327.685434] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7329.314677] Key type lgssc registered [ 7330.267945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7336.497803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7336.514674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.969897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.355863] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7347.171744] LustreError: 261778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7347.184522] LustreError: 261778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7349.010412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7349.286972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7352.429515] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7357.688643] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7361.687738] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7366.696282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.960976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.983723] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7377.897670] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7382.324100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7402.978778] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7402.984191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7403.004102] Lustre: Skipped 1 previous similar message [ 7403.033921] Lustre: Skipped 1 previous similar message [ 7408.099147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7408.102746] Lustre: Skipped 1 previous similar message [ 7409.166577] Lustre: server umount lustre-OST0000 complete [ 7413.217093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.223178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.235913] Lustre: Skipped 1 previous similar message [ 7418.337414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7418.340702] Lustre: Skipped 2 previous similar messages [ 7418.525923] Lustre: server umount lustre-MDT0000 complete [ 7421.757347] LustreError: 262513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787791993 with bad export cookie 16508620814159221281 [ 7421.760355] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.768827] LustreError: 262513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7422.032806] Lustre: server umount lustre-MDT0001 complete [ 7432.118467] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7434.545863] Key type lgssc unregistered [ 7434.902935] LNet: 266084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7434.927143] LNetError: 266084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7434.956126] LNet: Removed LNI 192.168.204.122@tcp [ 7435.695295] Key type .llcrypt unregistered [ 7435.697858] Key type ._llcrypt unregistered [ 7450.035984] Key type ._llcrypt registered [ 7450.040552] Key type .llcrypt registered [ 7450.132688] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7450.846839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7450.971133] alg: No test for adler32 (adler32-zlib) [ 7452.046645] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7452.296948] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7453.951169] Key type lgssc registered [ 7454.957493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7460.507574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7460.523625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7461.837742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.483755] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7471.078630] LustreError: 267287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.095745] LustreError: 267287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7472.463836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7472.766621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7476.059106] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7481.312745] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7485.418488] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7490.842549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.128893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.019456] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7502.045846] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7506.480387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7516.640534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7516.649216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.654029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.667698] Lustre: Skipped 1 previous similar message [ 7516.675546] Lustre: Skipped 1 previous similar message [ 7521.761918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.767896] Lustre: Skipped 1 previous similar message [ 7521.935618] Lustre: server umount lustre-OST0000 complete [ 7526.892500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.906073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7531.710556] Lustre: server umount lustre-MDT0000 complete [ 7532.001913] LustreError: 268400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7532.013745] LustreError: 268400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7535.221423] LustreError: 267267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792107 with bad export cookie 12975246793736650852 [ 7535.239244] LustreError: 267267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7535.240933] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.547488] Lustre: server umount lustre-MDT0001 complete [ 7545.887826] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7548.843936] Key type lgssc unregistered [ 7549.101482] LNet: 270965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7549.110543] LNetError: 270965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7549.131939] LNet: Removed LNI 192.168.204.122@tcp [ 7549.951891] Key type .llcrypt unregistered [ 7549.954417] Key type ._llcrypt unregistered [ 7569.612745] Key type ._llcrypt registered [ 7569.614768] Key type .llcrypt registered [ 7569.703414] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7571.102213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7571.141374] alg: No test for adler32 (adler32-zlib) [ 7572.264683] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7572.493312] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7574.143181] Key type lgssc registered [ 7575.202748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7581.704735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7581.736684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7583.361268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7587.176631] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7593.952388] LustreError: 272362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7593.961558] LustreError: 272362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7595.086262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.381305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7600.340777] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7606.507237] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7611.102081] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7616.222949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7616.492796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7622.515738] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7627.848746] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.780798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7656.422564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7656.439066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7656.459330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7659.502770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7659.508707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7662.529604] Lustre: server umount lustre-OST0000 complete [ 7667.188585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7667.201710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.218262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7667.224183] Lustre: Skipped 1 previous similar message [ 7669.734374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7669.742544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7673.675225] Lustre: server umount lustre-MDT0000 complete [ 7674.851220] LustreError: 272361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7674.884085] LustreError: 272361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7677.599579] LustreError: 272341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792249 with bad export cookie 3336283458444761863 [ 7677.603889] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7677.605610] LustreError: 272341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7677.766824] Lustre: server umount lustre-MDT0001 complete [ 7688.402857] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7691.063374] Key type lgssc unregistered [ 7691.340453] LNet: 276672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7691.356174] LNetError: 276672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7691.371661] LNet: Removed LNI 192.168.204.122@tcp [ 7692.153379] Key type .llcrypt unregistered [ 7692.155179] Key type ._llcrypt unregistered [ 7708.743538] Key type ._llcrypt registered [ 7708.745821] Key type .llcrypt registered [ 7708.875133] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7710.158518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7710.252108] alg: No test for adler32 (adler32-zlib) [ 7711.280569] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7711.661909] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7713.344301] Key type lgssc registered [ 7714.858761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7722.802429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7722.832958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.318801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.115088] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7734.756934] LustreError: 277910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.775485] LustreError: 277910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7735.936354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7736.316464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7739.825965] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7747.102310] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7754.198935] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7761.957047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7762.368955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.527693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7770.943147] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7779.849829] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7795.169911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.174669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.184395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7797.216220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7797.221564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7797.229994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.073123] Lustre: server umount lustre-OST0000 complete [ 7807.466593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7807.479463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7807.485126] Lustre: Skipped 1 previous similar message [ 7807.501754] Lustre: Skipped 2 previous similar messages [ 7810.294846] Lustre: server umount lustre-MDT0000 complete [ 7812.579347] LustreError: 277915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7812.600689] LustreError: 277915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7813.334535] LustreError: 277874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792385 with bad export cookie 4798700040453203788 [ 7813.338138] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7813.348599] LustreError: 277874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7813.615779] Lustre: server umount lustre-MDT0001 complete [ 7823.976569] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 7826.597459] Key type lgssc unregistered [ 7826.924895] LNet: 281616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7826.941430] LNetError: 281616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7826.966265] LNet: Removed LNI 192.168.204.122@tcp [ 7828.006735] Key type .llcrypt unregistered [ 7828.014426] Key type ._llcrypt unregistered [ 7839.008709] Key type ._llcrypt registered [ 7839.012567] Key type .llcrypt registered [ 7839.173820] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7854.819144] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 7868.717321] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 7870.180701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7870.231660] alg: No test for adler32 (adler32-zlib) [ 7871.368122] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 7871.618658] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 7873.319413] Key type lgssc registered [ 7874.192477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7912.963726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7912.984359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7914.220831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7914.256182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7914.330521] Lustre: lustre-MDT0000: new disk, initializing [ 7914.506379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.520905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7918.755849] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7929.476380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.614958] Lustre: 285179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7929.746962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7929.752207] Lustre: Skipped 1 previous similar message [ 7929.864903] Lustre: lustre-MDT0001: new disk, initializing [ 7930.016769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.055935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7930.071750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7934.765259] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7944.488317] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7950.533434] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7957.004154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.412344] Lustre: lustre-OST0000: new disk, initializing [ 7957.417278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7957.424143] Lustre: 287146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7957.521454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.234759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7964.257824] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7964.392273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7964.688356] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 7973.774263] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7978.192240] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7978.370558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7982.511761] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7982.725299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7985.126058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7985.135128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7985.137857] LustreError: Skipped 1 previous similar message [ 7985.145267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7985.159444] Lustre: Skipped 1 previous similar message [ 7990.245473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7990.255520] Lustre: Skipped 2 previous similar messages [ 7991.187554] Lustre: server umount lustre-OST0000 complete [ 7996.384856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7996.397990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.410281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8000.481193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8000.484843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.754703] Lustre: server umount lustre-MDT0000 complete [ 8005.601800] LustreError: 288187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.622471] LustreError: 288187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8005.663824] LustreError: 285171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792577 with bad export cookie 299165476383453449 [ 8005.669659] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8005.676704] LustreError: 285171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8006.242485] Lustre: server umount lustre-MDT0001 complete [ 8014.746214] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:03:06 (1787792586) [ 8020.935979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.614437] LustreError: 289559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8021.721498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8026.989684] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8027.107886] LustreError: 289560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8031.207353] LustreError: 289559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8035.951483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.440852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8041.190751] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8049.761897] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8055.789291] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.402467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8062.734647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8068.063579] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8074.379477] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8096.351469] Lustre: server umount lustre-OST0000 complete [ 8101.863282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8101.868904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8101.877504] Lustre: Skipped 1 previous similar message [ 8101.892928] Lustre: Skipped 2 previous similar messages [ 8110.140360] Lustre: server umount lustre-MDT0000 complete [ 8112.095124] Lustre: 283392:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787792668/real 1787792668] req@000000006ae03a38 x1874636248786048/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787792684 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8112.096881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8112.100163] LustreError: 289555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8112.131231] Lustre: 283392:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8113.812916] LustreError: 289540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792685 with bad export cookie 299165476383454331 [ 8113.824413] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8113.825083] LustreError: 289540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8114.132480] Lustre: server umount lustre-MDT0001 complete [ 8124.899433] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 8128.317880] Key type lgssc unregistered [ 8128.546231] LNet: 293124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8128.551038] LNetError: 293124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8128.567267] LNet: Removed LNI 192.168.204.122@tcp [ 8129.615842] Key type .llcrypt unregistered [ 8129.618067] Key type ._llcrypt unregistered [ 8167.333469] Key type ._llcrypt registered [ 8167.335679] Key type .llcrypt registered [ 8167.444524] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 8180.722098] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8181.935105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8182.092551] alg: No test for adler32 (adler32-zlib) [ 8183.294803] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 8183.630277] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 8185.391154] Key type lgssc registered [ 8186.594602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8242.456691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8242.482756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.785460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8243.803833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8243.881454] Lustre: lustre-MDT0000: new disk, initializing [ 8244.042432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.079934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8249.689480] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8261.458957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.623381] Lustre: 296950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8261.714428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8261.721528] Lustre: Skipped 1 previous similar message [ 8261.857336] Lustre: lustre-MDT0001: new disk, initializing [ 8261.998138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8262.040440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8262.056256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8266.873379] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8276.230418] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8281.875328] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8290.163551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8290.419307] Lustre: lustre-OST2710: new disk, initializing [ 8290.429708] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8290.434428] Lustre: 298918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8290.570335] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8296.086901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8296.105796] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8296.278768] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8298.826809] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8311.691494] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8322.562750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8322.812976] Lustre: lustre-OST03e8: new disk, initializing [ 8322.816223] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8322.821356] Lustre: 299983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8322.890990] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8329.841873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8329.852936] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8329.963885] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8330.708627] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8339.884045] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8347.963522] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8348.101836] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8353.214413] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8353.448629] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8358.653510] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8359.026605] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8364.575763] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8365.019608] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8371.173043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.181570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8371.199172] Lustre: Skipped 3 previous similar messages [ 8371.226853] Lustre: Skipped 3 previous similar messages [ 8372.646821] Lustre: server umount lustre-MDT0000 complete [ 8378.513689] LustreError: 298917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787792950 with bad export cookie 5310205480322196614 [ 8378.532644] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.866130] Lustre: server umount lustre-MDT0001 complete [ 8394.467377] Lustre: server umount lustre-OST2710 complete [ 8397.791187] Lustre: 294749:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787792953/real 1787792953] req@000000002ce72cc3 x1874636575941120/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787792969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8397.823566] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8399.734302] Lustre: server umount lustre-OST03e8 complete [ 8419.329400] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 8429.852697] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8471.368960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.688442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8471.717790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8471.792734] Lustre: lustre-MDT0000: new disk, initializing [ 8471.934470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.976779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8477.207734] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8488.767161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8488.861789] Lustre: 305236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8488.938178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8488.951692] Lustre: Skipped 1 previous similar message [ 8489.062750] Lustre: lustre-MDT0001: new disk, initializing [ 8489.219666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8489.273623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8489.298763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8494.747585] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8504.825879] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8511.716567] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8520.061945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8520.506928] Lustre: lustre-OST0000: new disk, initializing [ 8520.510554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8520.517640] Lustre: 307208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8520.665981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8527.965533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8527.985981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8528.306852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8529.012772] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8540.368678] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8545.702933] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8545.962505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8550.856983] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8551.019121] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8553.955855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8553.957672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8553.970048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8553.979439] Lustre: Skipped 1 previous similar message [ 8554.006484] Lustre: Skipped 1 previous similar message [ 8559.079417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8559.103587] Lustre: Skipped 1 previous similar message [ 8564.203659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8564.228108] Lustre: Skipped 1 previous similar message [ 8567.263135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8567.631065] Lustre: server umount lustre-OST0000 complete [ 8574.432791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8574.487736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8574.503474] Lustre: Skipped 1 previous similar message [ 8580.856858] Lustre: server umount lustre-MDT0000 complete [ 8584.688134] LustreError: 306257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8584.725714] LustreError: 306257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8586.124494] LustreError: 305228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793158 with bad export cookie 5310205480322198287 [ 8586.129191] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8586.137723] LustreError: 305228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8586.574812] Lustre: server umount lustre-MDT0001 complete [ 8597.388401] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:12:48 (1787793168) [ 8599.294588] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8602.116393] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:12:52 (1787793172) [ 8606.243253] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8638.053108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.487887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8638.517693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8638.782781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8644.851703] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8655.813567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8656.047789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8656.107755] Lustre: 311073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8661.691885] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8685.757695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8686.073294] Lustre: Denying initial registration attempt from nid 192.168.204.122@tcp, specified as failover [ 8686.092369] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8686.102356] LustreError: 312526:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8686.108476] LustreError: 312526:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8686.117493] LustreError: 312526:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8686.178225] Lustre: server umount lustre-OST0000 complete [ 8686.180402] LustreError: 312526:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8697.315212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8697.329946] LustreError: Skipped 1 previous similar message [ 8697.334281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.347107] Lustre: Skipped 1 previous similar message [ 8697.352969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8697.363347] Lustre: Skipped 2 previous similar messages [ 8700.804737] Lustre: server umount lustre-MDT0000 complete [ 8703.458559] LustreError: 311079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8703.474116] LustreError: 311079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8705.466369] LustreError: 311065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793277 with bad export cookie 5310205480322199176 [ 8705.480258] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.482622] LustreError: 311065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8705.983806] Lustre: server umount lustre-MDT0001 complete [ 8718.184582] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 8721.161551] Key type lgssc unregistered [ 8721.423787] LNet: 313654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8721.430451] LNetError: 313654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8721.455953] LNet: Removed LNI 192.168.204.122@tcp [ 8722.673676] Key type .llcrypt unregistered [ 8722.678637] Key type ._llcrypt unregistered [ 8736.902158] Key type ._llcrypt registered [ 8736.904604] Key type .llcrypt registered [ 8737.066490] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8738.003675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8738.020443] alg: No test for adler32 (adler32-zlib) [ 8739.103698] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 8739.421113] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 8741.167387] Key type lgssc registered [ 8742.681245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8793.363831] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8803.502689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8803.515799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.710767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8804.730758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8804.931273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.636324] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8819.203762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.351244] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8819.404287] Lustre: 316419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8819.779360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8825.241994] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8834.259798] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8840.829536] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8861.424744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.796133] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8862.096551] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8868.931764] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8877.694275] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8880.610232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8880.622845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8880.636160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8881.127275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8881.146988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.165521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8884.202282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.316379] Lustre: server umount lustre-OST0000 complete [ 8891.365223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8891.385472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.432756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8891.442735] Lustre: Skipped 1 previous similar message [ 8894.433710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8896.764262] Lustre: server umount lustre-MDT0000 complete [ 8901.403053] LustreError: 316412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793473 with bad export cookie 9975708606775444743 [ 8901.404108] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8901.421071] LustreError: 316412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8901.975099] Lustre: server umount lustre-MDT0001 complete [ 8912.933432] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 8915.349381] Key type lgssc unregistered [ 8915.596716] LNet: 319996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8915.601137] LNetError: 319996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8915.615547] LNet: Removed LNI 192.168.204.122@tcp [ 8916.518408] Key type .llcrypt unregistered [ 8916.521654] Key type ._llcrypt unregistered [ 8943.580750] Key type ._llcrypt registered [ 8943.582541] Key type .llcrypt registered [ 8943.682268] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 8944.784204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8944.820787] alg: No test for adler32 (adler32-zlib) [ 8945.856262] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 8946.167891] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 8947.919688] Key type lgssc registered [ 8949.302788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8957.315553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8957.329600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8958.833994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8963.933457] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8968.170848] LustreError: 321414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.204703] LustreError: 321414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8973.280721] LustreError: 321413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8973.658616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.979386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8978.688980] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 8987.424065] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8994.370228] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9004.843741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9005.294122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.520139] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9021.191241] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9040.354751] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9040.356328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9040.367508] Lustre: Skipped 1 previous similar message [ 9042.726771] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [ 9042.735360] Lustre: Skipped 2 previous similar messages [ 9045.473257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9045.485832] Lustre: Skipped 2 previous similar messages [ 9045.718864] Lustre: server umount lustre-MDT0000 complete [ 9047.817522] LustreError: 323230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.582372] LustreError: 321396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793621 with bad export cookie 7415300150013795253 [ 9049.585038] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.590441] LustreError: 321396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9050.598389] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.600257] LustreError: 321408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.607085] Lustre: Skipped 1 previous similar message [ 9050.609592] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9054.625364] LustreError: 323230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9054.625758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9054.652800] LustreError: 323230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9054.662212] Lustre: Skipped 1 previous similar message [ 9056.017534] Lustre: server umount lustre-MDT0001 complete [ 9063.886816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9073.956919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.159958] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x66e87032d0793991 [ 9074.173059] Lustre: MGC192.168.204.122@tcp: Connection restored to 0@lo (at 0@lo) [ 9074.474308] LustreError: 323246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9074.542748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9074.613836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9079.585844] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9084.777611] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9089.047243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9089.415263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9093.558483] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9094.648814] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9102.257727] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9103.645988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9110.028501] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9111.680769] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9120.232614] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9120.234263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9120.260362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9120.282974] Lustre: Skipped 1 previous similar message [ 9123.565586] Lustre: server umount lustre-OST0000 complete [ 9130.467387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.470248] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9130.492525] Lustre: Skipped 2 previous similar messages [ 9140.710791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9140.718144] Lustre: Skipped 5 previous similar messages [ 9142.239668] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9142.438265] Lustre: server umount lustre-MDT0000 complete [ 9145.828598] LustreError: 326279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9145.848380] LustreError: 326279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9146.242065] LustreError: 321396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793718 with bad export cookie 7415300150013802897 [ 9146.250269] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9146.262889] LustreError: 321396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9146.799682] Lustre: server umount lustre-MDT0001 complete [ 9159.190140] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 9162.415328] Key type lgssc unregistered [ 9162.816366] LNet: 327712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9162.830713] LNetError: 327712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9162.858903] LNet: Removed LNI 192.168.204.122@tcp [ 9163.936566] Key type .llcrypt unregistered [ 9163.939121] Key type ._llcrypt unregistered [ 9187.643908] Key type ._llcrypt registered [ 9187.645340] Key type .llcrypt registered [ 9187.719978] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9188.848449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9188.878165] alg: No test for adler32 (adler32-zlib) [ 9190.025393] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 9190.327783] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 9192.055135] Key type lgssc registered [ 9193.238485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9201.217475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9201.263705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.681679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9207.557829] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9212.899086] LustreError: 329130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.918498] LustreError: 329130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9216.040525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.440417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9221.538789] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9229.551328] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9237.397273] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9247.200554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9247.219548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9247.260410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9248.225419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9248.230342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.029137] Lustre: server umount lustre-MDT0000 complete [ 9253.346078] LustreError: 329858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.358245] LustreError: 329858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9254.039644] LustreError: 329111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793826 with bad export cookie 11802470724349249521 [ 9254.041884] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9254.051798] LustreError: 329111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9254.478914] Lustre: server umount lustre-MDT0001 complete [ 9271.175514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9271.533794] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9271.558847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9271.792596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.328386] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9286.801332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.881415] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9286.914297] Lustre: 331722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9287.174967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9291.918758] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9299.730256] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9305.733768] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9316.146803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9316.446980] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9316.465406] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9316.478981] LustreError: 333607:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9316.490151] LustreError: 333607:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9316.501596] LustreError: 333607:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9316.641445] Lustre: server umount lustre-OST0000 complete [ 9316.643719] LustreError: 333607:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9327.678660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9328.082979] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9328.522883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9336.435166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9336.467394] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9346.310546] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9355.751334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9355.850391] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9355.933410] Lustre: lustre-OST0001: new disk, initializing [ 9355.937234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9355.944039] Lustre: 335147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9356.038330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9361.767693] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9365.027342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9365.034996] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9365.147854] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9370.013281] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9375.209400] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9375.212810] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.238676] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9375.243756] Lustre: Skipped 2 previous similar messages [ 9379.044838] Lustre: server umount lustre-OST0001 complete [ 9380.337426] LustreError: 334075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.352044] LustreError: 334075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9382.889292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9382.896812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9382.907447] Lustre: Skipped 1 previous similar message [ 9382.914074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9385.448526] LustreError: 334074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9385.473660] LustreError: 334074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9389.014683] Lustre: server umount lustre-OST0000 complete [ 9394.656774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9394.664721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9394.675045] Lustre: Skipped 1 previous similar message [ 9394.680639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9394.687596] Lustre: Skipped 2 previous similar messages [ 9399.657696] Lustre: server umount lustre-MDT0000 complete [ 9400.806094] LustreError: 331727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9400.831362] LustreError: 331727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9403.840218] LustreError: 331712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787793975 with bad export cookie 11802470724349250074 [ 9403.852369] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9403.858342] LustreError: 331712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9404.305360] Lustre: server umount lustre-MDT0001 complete [ 9416.336681] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [ 9419.588837] Key type lgssc unregistered [ 9419.921812] LNet: 337109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9419.938902] LNetError: 337109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9420.967465] LNet: Removed LNI 192.168.204.122@tcp [ 9422.085118] Key type .llcrypt unregistered [ 9422.089257] Key type ._llcrypt unregistered [ 9498.468613] Key type ._llcrypt registered [ 9498.471544] Key type .llcrypt registered [ 9498.574924] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [ 9514.725576] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [ 9516.414462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9516.492135] alg: No test for adler32 (adler32-zlib) [ 9517.651366] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [ 9517.852971] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [ 9519.544110] Key type lgssc registered [ 9520.888980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9564.107973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9564.162810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.558403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9565.621399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9565.736281] Lustre: lustre-MDT0000: new disk, initializing [ 9565.887245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.914556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9571.405407] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9583.620102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.736103] Lustre: 342062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9583.938122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9583.945874] Lustre: Skipped 1 previous similar message [ 9584.142437] Lustre: lustre-MDT0001: new disk, initializing [ 9584.293344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9584.355360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9584.377698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9589.542044] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9600.417028] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9608.231275] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9616.641071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9616.998434] Lustre: lustre-OST0000: new disk, initializing [ 9617.006920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9617.024469] Lustre: 344033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9617.177398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9624.216373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9624.238752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9624.309337] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9624.404943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9634.643791] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9639.383534] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9639.619913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9644.729804] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9644.951382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9650.152523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9650.157330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.158863] LustreError: Skipped 1 previous similar message [ 9650.174701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.181253] Lustre: Skipped 1 previous similar message [ 9655.277305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9655.289246] Lustre: Skipped 1 previous similar message [ 9660.389887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9660.397443] Lustre: Skipped 1 previous similar message [ 9660.897128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9661.084335] Lustre: server umount lustre-OST0000 complete [ 9665.507932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9665.515699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9665.529390] Lustre: Skipped 2 previous similar messages [ 9670.630600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9670.646414] Lustre: Skipped 2 previous similar messages [ 9670.794323] Lustre: server umount lustre-MDT0000 complete [ 9674.339724] LustreError: 342055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794246 with bad export cookie 10652821792271677915 [ 9674.348375] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9674.638614] Lustre: server umount lustre-MDT0001 complete [ 9682.275063] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 21:30:54 (1787794254) [ 9689.281351] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 21:31:00 (1787794260) [ 9697.872800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.414440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9702.991525] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9703.904372] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9703.913034] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9709.025848] LustreError: 346790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9711.281218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9711.563189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9715.930848] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9723.596529] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9728.717692] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9734.909817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.309798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.337339] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9749.785907] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9758.520563] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9760.002684] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9761.920964] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9763.874465] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9765.645870] Lustre: Failing over lustre-MDT0000 [ 9766.039444] Lustre: server umount lustre-MDT0000 complete [ 9767.905712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9767.912770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.934237] LustreError: 346790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9770.464940] LustreError: 347517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9770.468193] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.496116] LustreError: 347517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9775.596174] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9775.608732] LustreError: 346789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9784.652933] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9784.687793] LustreError: 346785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9785.899494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9786.077559] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.484955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.564946] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9789.689188] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9791.980358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9792.036243] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9792.102367] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9792.111087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9802.958909] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9805.051211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9809.176453] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9811.459165] Lustre: Failing over lustre-MDT0000 [ 9812.154326] Lustre: server umount lustre-MDT0000 complete [ 9812.454080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9812.458333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.473569] LustreError: 350524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.473580] LustreError: 350524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9812.521561] Lustre: Skipped 3 previous similar messages [ 9828.832131] Lustre: 340288:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787794384/real 1787794384] req@000000006724d782 x1874637974761216/t0(0) o400->MGC192.168.204.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787794400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9828.874764] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.098131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.503574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9838.586232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9840.385511] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9843.707852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9843.721487] Lustre: Skipped 2 previous similar messages [ 9843.749853] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9843.810469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9844.081834] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9854.868268] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9856.681301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9864.167273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9864.168039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9864.174235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9864.178178] Lustre: Skipped 2 previous similar messages [ 9869.517488] Lustre: server umount lustre-MDT0000 complete [ 9873.182487] LustreError: 346770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794445 with bad export cookie 10652821792271680624 [ 9873.195223] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9873.196545] LustreError: 346770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9873.517409] Lustre: server umount lustre-MDT0001 complete [ 9887.655328] Lustre: server umount lustre-OST0000 complete [ 9904.402396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.951985] LustreError: 353131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9904.977000] LustreError: 353131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 9905.060150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9910.582728] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9921.903706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.736326] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9938.579145] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9945.027380] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9953.341559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.005862] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [ 9968.902079] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9974.196740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9983.471573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9983.487277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9983.532543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9983.551154] Lustre: Skipped 5 previous similar messages [ 9995.231167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9995.554769] Lustre: server umount lustre-OST0000 complete [10004.448581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10004.461559] LustreError: Skipped 1 previous similar message [10006.767231] Lustre: server umount lustre-MDT0000 complete [10010.082083] LustreError: 353127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10010.110142] LustreError: 353127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10010.997082] LustreError: 353113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794582 with bad export cookie 10652821792271681373 [10011.001867] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.008607] LustreError: 353113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10011.276938] Lustre: server umount lustre-MDT0001 complete [10025.376914] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [10028.395423] Key type lgssc unregistered [10028.743911] LNet: 356671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10028.754357] LNetError: 356671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10028.780838] LNet: Removed LNI 192.168.204.122@tcp [10029.972829] Key type .llcrypt unregistered [10029.975375] Key type ._llcrypt unregistered [10058.631848] Key type ._llcrypt registered [10058.634304] Key type .llcrypt registered [10058.739540] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10059.796847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10059.861910] alg: No test for adler32 (adler32-zlib) [10061.079945] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [10061.334313] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [10062.968639] Key type lgssc registered [10064.239442] Lustre: Echo OBD driver; http://www.lustre.org/ [10073.562310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10073.578857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.011598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10080.040874] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10084.320875] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.343083] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10089.439957] LustreError: 358192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10090.599085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10091.196616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10096.850524] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10106.770793] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10113.754618] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10121.641052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.054465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10129.689504] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10138.551453] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10144.568328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10160.112953] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.119804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.136863] Lustre: Skipped 1 previous similar message [10160.152111] Lustre: Skipped 2 previous similar messages [10161.403867] Lustre: server umount lustre-MDT0000 complete [10165.220789] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10165.244729] LustreError: 358191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10165.642811] LustreError: 358172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794737 with bad export cookie 3122196764724731860 [10165.653138] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10166.010610] Lustre: server umount lustre-MDT0001 complete [10180.246273] Lustre: server umount lustre-OST0000 complete [10195.963541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.541219] LustreError: 361933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10196.666067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10201.550750] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10202.090581] LustreError: 361934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.007936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.299596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10211.439670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10215.612763] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10222.932511] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10229.098942] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10236.945968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10237.313788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10241.400765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10243.684277] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10251.083376] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10267.115456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.118786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10267.133918] Lustre: Skipped 3 previous similar messages [10267.154801] Lustre: Skipped 2 previous similar messages [10269.020243] Lustre: server umount lustre-MDT0000 complete [10272.226853] LustreError: 364568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10272.255806] LustreError: 364568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10272.910422] LustreError: 361913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794844 with bad export cookie 3122196764724732973 [10272.913334] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10272.921972] LustreError: 361913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10273.348950] Lustre: server umount lustre-MDT0001 complete [10287.663298] Lustre: server umount lustre-OST0000 complete [10304.387729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10304.952807] LustreError: 365842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10305.075703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10309.158741] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10316.279847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.326323] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10328.791718] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10334.558785] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10340.719223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10340.975227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10340.981387] Lustre: Skipped 1 previous similar message [10346.801379] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10349.173576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10354.313334] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10359.777941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.785581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10359.796908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10359.808697] Lustre: Skipped 1 previous similar message [10364.900454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10364.904698] Lustre: Skipped 1 previous similar message [10365.065769] Lustre: server umount lustre-OST0000 complete [10370.016529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10370.024800] Lustre: Skipped 2 previous similar messages [10375.144222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10375.147649] Lustre: Skipped 3 previous similar messages [10383.327155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10383.483473] Lustre: server umount lustre-MDT0000 complete [10385.378977] LustreError: 365843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.393639] LustreError: 365843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10387.148525] LustreError: 365824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787794959 with bad export cookie 3122196764724733988 [10387.157919] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.162160] LustreError: 365824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10387.508310] Lustre: server umount lustre-MDT0001 complete [10399.340194] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [10402.400081] Key type lgssc unregistered [10402.783809] LNet: 369376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10402.793874] LNetError: 369376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10402.823119] LNet: Removed LNI 192.168.204.122@tcp [10403.949652] Key type .llcrypt unregistered [10403.951472] Key type ._llcrypt unregistered [10430.595541] Key type ._llcrypt registered [10430.601126] Key type .llcrypt registered [10430.715064] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10431.750256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10431.846687] alg: No test for adler32 (adler32-zlib) [10432.892500] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [10433.131490] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [10434.863181] Key type lgssc registered [10435.970195] Lustre: Echo OBD driver; http://www.lustre.org/ [10443.981805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10444.013168] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10444.033301] LustreError: 370821:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10444.056564] LustreError: 370821:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10444.072399] LustreError: 370821:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10444.084877] LustreError: 370821:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10444.099886] LustreError: 370821:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10452.757954] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10452.775393] LustreError: 371218:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10452.795450] LustreError: 371218:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10452.803205] LustreError: 371218:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10452.811900] LustreError: 371218:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10452.823826] LustreError: 371218:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10473.484069] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [10476.571170] Key type lgssc unregistered [10476.848328] LNet: 372225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10476.860234] LNetError: 372225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10476.881492] LNet: Removed LNI 192.168.204.122@tcp [10477.774406] Key type .llcrypt unregistered [10477.779924] Key type ._llcrypt unregistered [10498.343946] Key type ._llcrypt registered [10498.348163] Key type .llcrypt registered [10498.464340] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [10513.291306] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10514.696819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10514.708049] alg: No test for adler32 (adler32-zlib) [10515.877298] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [10516.141625] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [10517.831196] Key type lgssc registered [10518.953728] Lustre: Echo OBD driver; http://www.lustre.org/ [10557.115224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10557.166944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10558.458500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10558.501917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10558.606203] Lustre: lustre-MDT0000: new disk, initializing [10558.752485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10558.796077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10563.562767] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10573.990684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.085632] Lustre: 375381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10574.160857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10574.163952] Lustre: Skipped 1 previous similar message [10574.217710] Lustre: lustre-MDT0001: new disk, initializing [10574.299505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10574.334469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10574.348665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10578.997914] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10589.010415] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10596.393765] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10604.983743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10605.320856] Lustre: lustre-OST0000: new disk, initializing [10605.325296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10605.335949] Lustre: 377355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10605.441909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10612.886818] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10615.342739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10615.358885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10615.442836] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10622.968852] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10627.560957] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10627.786887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10632.323540] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10632.486256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10636.258863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10636.261996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.277758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10636.289096] Lustre: Skipped 1 previous similar message [10636.304628] Lustre: Skipped 1 previous similar message [10640.791658] Lustre: server umount lustre-OST0000 complete [10645.987373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10645.998905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10646.015298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.492762] Lustre: server umount lustre-MDT0000 complete [10651.623862] LustreError: 375394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10651.649354] LustreError: 375394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10654.282088] LustreError: 375374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795226 with bad export cookie 2633149262721814191 [10654.284548] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10654.296385] LustreError: 375374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10654.746902] Lustre: server umount lustre-MDT0001 complete [10663.079212] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10665.071433] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 21:47:16 (1787795236) [10671.539789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.027125] LustreError: 379813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10672.108045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10675.911381] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10677.223959] LustreError: 379814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10682.338690] LustreError: 379813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.288129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10684.740890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10689.104258] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10696.257390] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10701.296992] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10708.002641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.311524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10714.808830] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10721.881970] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10728.759432] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10728.942361] Lustre: lustre-OST0001: new disk, initializing [10728.953076] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10728.963386] Lustre: 382648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10729.113319] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10734.712421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10734.725747] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10734.858625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10737.893555] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10750.245694] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10755.582682] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10755.590304] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10755.603945] LustreError: Skipped 1 previous similar message [10755.606180] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10755.622252] Lustre: Skipped 2 previous similar messages [10755.628739] Lustre: Skipped 2 previous similar messages [10760.412283] Lustre: server umount lustre-OST0001 complete [10760.680217] LustreError: 381634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.690696] LustreError: 381634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10770.919224] LustreError: 381636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.942084] LustreError: 381636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10779.105972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10779.111076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10779.134911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10779.138835] Lustre: Skipped 2 previous similar messages [10784.265561] Lustre: server umount lustre-OST0000 complete [10791.396223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.398265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10791.409790] Lustre: Skipped 2 previous similar messages [10791.425587] Lustre: Skipped 3 previous similar messages [10794.806104] Lustre: server umount lustre-MDT0000 complete [10796.531073] LustreError: 380542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10796.561696] LustreError: 380542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10799.323303] LustreError: 382647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795371 with bad export cookie 2633149262721815073 [10799.346888] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10799.354606] LustreError: 382647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10805.893595] Lustre: server umount lustre-MDT0001 complete [10818.042636] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [10820.879992] Key type lgssc unregistered [10821.187592] LNet: 384643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10821.196213] LNetError: 384643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10821.209713] LNet: Removed LNI 192.168.204.122@tcp [10821.975161] Key type .llcrypt unregistered [10821.977615] Key type ._llcrypt unregistered [10875.450916] Key type ._llcrypt registered [10875.454642] Key type .llcrypt registered [10875.582958] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [10876.813878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10876.871087] alg: No test for adler32 (adler32-zlib) [10878.022130] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [10878.297749] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [10880.007751] Key type lgssc registered [10881.295861] Lustre: Echo OBD driver; http://www.lustre.org/ [10889.946628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10889.966022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10891.265738] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10891.294764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10891.466973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10895.847323] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10904.242370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.358556] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10904.406987] Lustre: 387442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10904.669330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10909.427281] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10918.460745] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10924.186657] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10930.145371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10930.160494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.182350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10930.660576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10930.683639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10932.697610] Lustre: server umount lustre-MDT0000 complete [10937.042532] LustreError: 387435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795509 with bad export cookie 2480738369725980190 [10937.043703] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10937.059368] LustreError: 387435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10937.478354] Lustre: server umount lustre-MDT0001 complete [10947.073806] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 21:51:58 (1787795518) [10954.416969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10954.936225] LustreError: 389954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.948069] LustreError: 389954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10954.991135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10959.805541] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10960.357174] LustreError: 389955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10965.480403] LustreError: 389954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.307221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10969.895236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10975.161130] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [10984.061883] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10989.840821] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10997.164854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10997.259599] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10997.403385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11004.189336] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11012.244456] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11022.278584] Lustre: Permanently deactivating lustre-OST0000 [11022.301487] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11027.516577] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11029.001322] LustreError: 392845:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11029.014542] LustreError: 392845:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11029.023313] LustreError: 392845:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11033.272626] LustreError: 392893:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11033.281709] LustreError: 392893:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11041.616588] Lustre: server umount lustre-OST0000 complete [11044.565887] LustreError: 393144:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11044.570406] LustreError: 393144:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11046.884966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11046.899452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11046.920522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11049.441275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11049.442974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11054.561220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11054.577272] Lustre: Skipped 2 previous similar messages [11056.695736] Lustre: server umount lustre-MDT0000 complete [11059.681540] LustreError: 390935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.700204] LustreError: 390935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11061.007417] LustreError: 389934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795632 with bad export cookie 2480738369725980771 [11061.010115] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11061.019754] LustreError: 389934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11071.670640] Lustre: server umount lustre-MDT0001 complete [11081.442918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.666673] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11089.734544] LustreError: 394302:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11096.772042] Lustre: 394641:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.122@tcp [11099.713633] Lustre: server umount MGS complete [11111.573186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.861197] LustreError: 395367:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11111.873881] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11111.876609] Lustre: Skipped 1 previous similar message [11111.918319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11116.033098] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11122.155518] LustreError: 395368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.174981] LustreError: 395368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11125.094676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.338833] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11130.339268] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11139.229983] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11147.030673] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11149.703711] Lustre: Permanently reactivating lustre-OST0000 [11149.746614] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11149.760403] Lustre: Skipped 2 previous similar messages [11176.928890] LustreError: 397102:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11184.190123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11184.692618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11184.697073] Lustre: Skipped 1 previous similar message [11192.029488] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11200.439385] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11207.091246] Lustre: server umount lustre-OST0000 complete [11212.262574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11212.276128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11212.283206] Lustre: Skipped 1 previous similar message [11212.306498] Lustre: Skipped 1 previous similar message [11221.566587] Lustre: server umount lustre-MDT0000 complete [11222.502255] LustreError: 395364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11222.523248] LustreError: 395364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11223.520532] Lustre: 386767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787795779/real 1787795779] req@00000000920e876f x1874639401876992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787795795 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11223.568230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.635659] LustreError: 395347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787795798 with bad export cookie 2480738369725981821 [11226.637461] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11226.665410] LustreError: 395347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11227.070772] Lustre: server umount lustre-MDT0001 complete [11240.653812] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11243.893646] Key type lgssc unregistered [11244.204227] LNet: 399689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11244.216046] LNetError: 399689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11244.249206] LNet: Removed LNI 192.168.204.122@tcp [11245.148770] Key type .llcrypt unregistered [11245.153677] Key type ._llcrypt unregistered [11266.909478] Key type ._llcrypt registered [11266.912530] Key type .llcrypt registered [11267.051068] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [11287.327914] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11288.666436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11289.148098] alg: No test for adler32 (adler32-zlib) [11290.715704] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [11291.334616] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11293.167156] Key type lgssc registered [11295.064140] Lustre: Echo OBD driver; http://www.lustre.org/ [11339.227402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11339.259436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11340.481730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11340.525763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11340.661849] Lustre: lustre-MDT0000: new disk, initializing [11340.813451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11340.840564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11346.793991] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11359.905231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11360.097888] Lustre: 402848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11360.236784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11360.240222] Lustre: Skipped 1 previous similar message [11360.360668] Lustre: lustre-MDT0001: new disk, initializing [11360.490969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11360.545839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11360.566886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11366.045052] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11376.406353] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.130414] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11390.669703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11391.028808] Lustre: lustre-OST0000: new disk, initializing [11391.039694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11391.070709] Lustre: 404820:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11391.218630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11398.858570] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11400.778755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11400.790615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11400.901049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11409.591943] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11415.322512] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11415.511691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11421.542413] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11421.808866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11426.788371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11426.789352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11426.798306] LustreError: Skipped 1 previous similar message [11426.801053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11426.821968] Lustre: Skipped 1 previous similar message [11426.842934] Lustre: Skipped 1 previous similar message [11430.206269] Lustre: server umount lustre-OST0000 complete [11437.027596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.056346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11441.048268] Lustre: server umount lustre-MDT0000 complete [11445.342904] LustreError: 402842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796017 with bad export cookie 12900778501614105547 [11445.348871] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11445.363025] LustreError: 402842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11445.837789] Lustre: server umount lustre-MDT0001 complete [11456.282728] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11458.253612] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:00:29 (1787796029) [11465.726473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11466.173328] LustreError: 407282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11466.202954] LustreError: 407282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11466.249162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11470.859709] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11471.331352] LustreError: 407283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11476.449343] LustreError: 407282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.089932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11481.546588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11486.563515] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11496.082935] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11503.004747] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11511.337903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11511.734434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11518.917994] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11528.355289] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11530.474606] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11537.478955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11546.598869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11546.603324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11546.608676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11546.608685] Lustre: Skipped 1 previous similar message [11546.644490] Lustre: Skipped 2 previous similar messages [11546.697903] LustreError: Skipped 1 previous similar message [11550.878875] Lustre: server umount lustre-OST0000 complete [11556.838727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11556.861242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11556.868524] Lustre: Skipped 1 previous similar message [11561.818718] Lustre: server umount lustre-MDT0000 complete [11561.955775] LustreError: 408012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.976566] LustreError: 408012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11566.059821] LustreError: 407283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.089291] LustreError: 407283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11566.237969] LustreError: 407264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796138 with bad export cookie 12900778501614106429 [11566.268186] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.582417] Lustre: server umount lustre-MDT0001 complete [11578.779617] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11581.854394] Key type lgssc unregistered [11582.170089] LNet: 411027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11582.180344] LNetError: 411027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11582.205387] LNet: Removed LNI 192.168.204.122@tcp [11583.148923] Key type .llcrypt unregistered [11583.150932] Key type ._llcrypt unregistered [11610.846930] Key type ._llcrypt registered [11610.855829] Key type .llcrypt registered [11610.995784] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11611.306424] Key type .llcrypt unregistered [11611.310619] Key type ._llcrypt unregistered [11630.404515] Key type ._llcrypt registered [11630.408681] Key type .llcrypt registered [11630.576736] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11632.213342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11632.253634] alg: No test for adler32 (adler32-zlib) [11633.526936] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [11633.884324] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11635.583185] Key type lgssc registered [11636.848332] Lustre: Echo OBD driver; http://www.lustre.org/ [11644.667248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11644.689628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.281106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11651.164613] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11656.679845] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.705757] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11660.552733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11661.100166] LustreError: 413246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.136546] LustreError: 413246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11661.153582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11666.412089] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11666.451565] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11670.068251] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11671.529244] LustreError: 413246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.558458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11671.580717] LustreError: 413246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11676.648730] LustreError: 413990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11676.687472] LustreError: 413990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11678.832044] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11686.891895] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11686.908297] LustreError: 413245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11687.575139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11688.073185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11693.755347] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11706.610757] Lustre: server umount lustre-OST0000 complete [11710.386663] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11710.396037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11710.421644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.017139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11714.020391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11719.138250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11719.153251] Lustre: Skipped 2 previous similar messages [11720.306949] Lustre: server umount lustre-MDT0000 complete [11724.258263] Lustre: 412557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787796280/real 1787796280] req@00000000b595a3cc x1874640193483008/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787796296 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11724.286364] LustreError: 413242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11724.319217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11724.333588] LustreError: 413242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11724.821348] LustreError: 413226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796296 with bad export cookie 3506284020794800882 [11724.837533] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11724.840031] LustreError: 413226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11725.518161] Lustre: server umount lustre-MDT0001 complete [11739.306477] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11742.731488] Key type lgssc unregistered [11743.311879] LNet: 416397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11743.322798] LNetError: 416397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11743.343875] LNet: Removed LNI 192.168.204.122@tcp [11744.383666] Key type .llcrypt unregistered [11744.391185] Key type ._llcrypt unregistered [11777.372918] Key type ._llcrypt registered [11777.375824] Key type .llcrypt registered [11777.607258] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11779.181354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11779.221181] alg: No test for adler32 (adler32-zlib) [11780.395244] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [11780.793834] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11782.673657] Key type lgssc registered [11783.856980] Lustre: Echo OBD driver; http://www.lustre.org/ [11793.922190] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11793.960270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.503635] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11814.623352] LustreError: 417788:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.122@tcp: failed processing log, type 4: rc = -110 [11841.311527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11848.368088] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11857.548715] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11864.826570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11865.397218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11865.504436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11870.692147] LustreError: 417814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11870.716093] LustreError: 417814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11871.088938] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11875.818550] LustreError: 417813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11876.840368] LustreError: 418827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.923462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11881.952493] LustreError: 417813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11881.972332] LustreError: 417813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11882.390279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.962976] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [11897.887538] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11905.694216] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11916.769731] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11916.781976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11916.798565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11918.305137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11918.306062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11918.334381] Lustre: Skipped 1 previous similar message [11923.435424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11923.441190] Lustre: Skipped 1 previous similar message [11928.032387] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11928.196647] Lustre: server umount lustre-OST0000 complete [11933.670486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.679113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11933.679272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11933.692268] Lustre: Skipped 1 previous similar message [11938.301259] Lustre: server umount lustre-MDT0000 complete [11938.791078] LustreError: 418828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11938.824151] LustreError: 418828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11942.385704] LustreError: 417794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796514 with bad export cookie 16122743213123331144 [11942.393995] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11942.407757] LustreError: 417794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11942.811136] Lustre: server umount lustre-MDT0001 complete [11956.398565] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [11959.873435] Key type lgssc unregistered [11960.315711] LNet: 421365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11960.330065] LNetError: 421365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11960.352403] LNet: Removed LNI 192.168.204.122@tcp [11961.517686] Key type .llcrypt unregistered [11961.524108] Key type ._llcrypt unregistered [11990.598426] Key type ._llcrypt registered [11990.600813] Key type .llcrypt registered [11990.703428] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [11991.664522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11991.781522] alg: No test for adler32 (adler32-zlib) [11992.926657] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [11993.246974] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [11994.919202] Key type lgssc registered [11996.431537] Lustre: Echo OBD driver; http://www.lustre.org/ [12004.452845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12004.471494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12005.862100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12011.221041] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12015.075940] LustreError: 422784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.105076] LustreError: 422784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12020.199202] LustreError: 422785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.030912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12021.411741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12026.834054] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12035.176755] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12041.740692] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12049.076210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12049.466390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12050.481627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12056.060438] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12065.210102] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12071.916237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12071.933122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12071.941044] Lustre: Skipped 1 previous similar message [12071.951582] Lustre: Skipped 1 previous similar message [12073.244030] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.22@tcp (stopping) [12073.262244] Lustre: Skipped 1 previous similar message [12076.643679] Lustre: server umount lustre-MDT0000 complete [12077.031904] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.056072] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12079.393260] LustreError: 422764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.22@tcp arrived at 1787796651 with bad export cookie 10078964028104096865 [12079.430167] LustreError: 422764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12081.675534] LustreError: 423518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787796653 with bad export cookie 10078964028104095976 [12081.690547] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12081.705024] LustreError: 423518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12082.144205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12082.147866] LustreError: 422780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12082.148466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12082.148474] Lustre: Skipped 1 previous similar message [12082.156136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12082.213289] LustreError: 422780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12086.176807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12086.177505] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12086.203912] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12088.079792] Lustre: server umount lustre-OST0000 complete [12095.545504] Lustre: server umount lustre-MDT0001 complete [12106.647138] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12110.034611] Key type lgssc unregistered [12110.421931] LNet: 426476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12110.434733] LNetError: 426476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12110.468815] LNet: Removed LNI 192.168.204.122@tcp [12111.645627] Key type .llcrypt unregistered [12111.647690] Key type ._llcrypt unregistered [12143.145720] Key type ._llcrypt registered [12143.154837] Key type .llcrypt registered [12143.315261] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12144.787920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12144.879749] alg: No test for adler32 (adler32-zlib) [12146.183782] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [12146.491598] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [12148.200526] Key type lgssc registered [12149.685392] Lustre: Echo OBD driver; http://www.lustre.org/ [12159.052610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12159.084027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12160.624776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12166.210118] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12170.215409] LustreError: 427894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.248637] LustreError: 427894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12175.331978] LustreError: 427895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12175.963821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.388107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12182.453954] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12192.319699] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12200.347352] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12208.390846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12208.873385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12215.896663] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12225.074926] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12230.399143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12235.753229] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12235.757602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12235.767845] Lustre: Skipped 1 previous similar message [12235.783724] Lustre: Skipped 1 previous similar message [12239.934668] Lustre: server umount lustre-MDT0001 complete [12240.868520] LustreError: 427889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12240.898653] LustreError: 427889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12245.990609] LustreError: 428622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.015357] LustreError: 428622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12251.107898] LustreError: 427894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12251.108540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12251.111630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.114313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12251.114321] Lustre: Skipped 1 previous similar message [12253.828977] Lustre: server umount lustre-OST0000 complete [12258.849047] Lustre: server umount lustre-MDT0000 complete [12277.152151] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12280.200948] Key type lgssc unregistered [12280.499987] LNet: 431586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12280.508949] LNetError: 431586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12280.524051] LNet: Removed LNI 192.168.204.122@tcp [12281.496363] Key type .llcrypt unregistered [12281.499051] Key type ._llcrypt unregistered [12312.771380] Key type ._llcrypt registered [12312.774115] Key type .llcrypt registered [12312.928711] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [12332.709338] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12333.871647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12333.913913] alg: No test for adler32 (adler32-zlib) [12335.121205] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [12335.384325] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [12337.058701] Key type lgssc registered [12338.682528] Lustre: Echo OBD driver; http://www.lustre.org/ [12382.954223] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12401.783990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12401.851928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12403.235353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12403.271527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12403.408469] Lustre: lustre-MDT0000: new disk, initializing [12403.532467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12403.562922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12409.269459] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12427.831350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12428.281022] Lustre: lustre-OST0000: new disk, initializing [12428.289809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12428.296225] Lustre: Skipped 1 previous similar message [12428.327210] Lustre: 436625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12428.441685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12434.062155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12434.075557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12434.111740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12436.454567] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12448.096485] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12465.942788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12466.138385] Lustre: 435603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12466.219833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12466.298788] Lustre: lustre-MDT0001: new disk, initializing [12466.400630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12466.437393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12466.449549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12471.731924] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12482.551416] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12482.796714] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12487.610659] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12487.803972] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12493.793404] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12493.796823] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12493.810903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12493.847997] Lustre: Skipped 1 previous similar message [12496.686671] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.22@tcp (stopping) [12498.916538] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12498.929591] Lustre: Skipped 1 previous similar message [12499.301926] Lustre: server umount lustre-MDT0001 complete [12509.168042] LustreError: 435611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.222039] LustreError: 435611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12514.290630] LustreError: 435610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12514.318576] LustreError: 435610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12519.397909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.404440] LustreError: 435616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.418930] Lustre: Skipped 1 previous similar message [12519.420887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12524.360740] Lustre: server umount lustre-OST0000 complete [12524.514303] LustreError: 435615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12534.726248] Lustre: server umount lustre-MDT0000 complete [12549.871557] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12552.713907] Key type lgssc unregistered [12553.015981] LNet: 440377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12553.025616] LNetError: 440377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12553.040722] LNet: Removed LNI 192.168.204.122@tcp [12553.867669] Key type .llcrypt unregistered [12553.874790] Key type ._llcrypt unregistered [12603.748466] Key type ._llcrypt registered [12603.754631] Key type .llcrypt registered [12603.964288] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [12604.438549] Key type .llcrypt unregistered [12604.450343] Key type ._llcrypt unregistered [12626.401333] Key type ._llcrypt registered [12626.406023] Key type .llcrypt registered [12626.652689] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [12628.385041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12628.589385] alg: No test for adler32 (adler32-zlib) [12630.275702] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [12630.536714] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [12632.175917] Key type lgssc registered [12633.577433] Lustre: Echo OBD driver; http://www.lustre.org/ [12668.391843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12668.415649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12669.850742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12669.905712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12670.025351] Lustre: lustre-MDT0000: new disk, initializing [12670.126279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12670.156940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12675.512689] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12689.522828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12689.731332] Lustre: 443760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12689.814992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12689.824565] Lustre: Skipped 1 previous similar message [12689.955755] Lustre: lustre-MDT0001: new disk, initializing [12690.158252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12690.200285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12690.214086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12695.889732] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12706.805931] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12714.023907] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12722.869176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12723.242781] Lustre: lustre-OST0000: new disk, initializing [12723.256614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12723.263357] Lustre: 445732:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12723.391962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12731.140563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12731.156916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12731.378405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12732.543817] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12746.166903] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12756.978996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12756.984176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12757.009554] Lustre: Skipped 2 previous similar messages [12757.012596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.301947] Lustre: server umount lustre-MDT0000 complete [12764.289677] LustreError: 443753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797336 with bad export cookie 4767833845756229408 [12764.292408] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12764.302695] LustreError: 443753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12764.646950] Lustre: server umount lustre-MDT0001 complete [12779.726484] Lustre: server umount lustre-OST0000 complete [12791.681316] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:22:43 (1787797363) [12799.485716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12799.972970] LustreError: 447763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.994988] LustreError: 447763:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12800.060793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12804.823138] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12805.094531] LustreError: 447764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12805.115297] LustreError: 447764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12809.195407] LustreError: 447763:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.305744] LustreError: 447764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12814.349299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12814.747519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12819.456260] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12829.037164] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12836.428344] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12857.841219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12858.115139] Lustre: Found index 0 for lustre-OST0000, updating log [12858.320309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12865.047167] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [12870.661508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12874.039969] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12880.869546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12880.871276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12880.876792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12880.876803] Lustre: Skipped 2 previous similar messages [12880.904995] Lustre: Skipped 1 previous similar message [12881.956983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12881.977240] Lustre: Skipped 2 previous similar messages [12886.516897] Lustre: server umount lustre-OST0000 complete [12903.400110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12903.409899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12903.423215] Lustre: Skipped 1 previous similar message [12903.439816] Lustre: Skipped 1 previous similar message [12908.517979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12908.538775] Lustre: Skipped 1 previous similar message [12909.537584] LustreError: 447764:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12909.561692] LustreError: 447764:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12909.633979] Lustre: server umount lustre-MDT0000 complete [12914.148747] LustreError: 450357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787797486 with bad export cookie 4767833845756230801 [12914.149672] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12914.161021] LustreError: 450357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12914.880016] Lustre: server umount lustre-MDT0001 complete [12918.303419] LNet: 1 local NIs in recovery (showing 1): 192.168.204.122@tcp [12925.979952] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:24:56 (1787797496) [13553.393765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13553.967537] LustreError: 454179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13553.986781] LustreError: 454179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13554.066070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13559.266071] LustreError: 454180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13559.299761] LustreError: 454180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13559.944706] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13563.362085] LustreError: 454179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13569.506189] LustreError: 454180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13569.521769] LustreError: 454180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13570.002444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13570.371340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13575.623870] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13583.382788] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13588.976940] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13609.531498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13609.830209] Lustre: Found index 0 for lustre-OST0000, updating log [13610.115937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13610.527803] LNet: 1 local NIs in recovery (showing 1): 192.168.204.122@tcp [13617.898534] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13626.531804] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13629.013899] Lustre: server umount lustre-OST0000 complete [13634.022066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13634.024356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13634.040892] Lustre: Skipped 1 previous similar message [13635.108710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13635.112895] Lustre: Skipped 1 previous similar message [13637.218593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13637.227690] Lustre: Skipped 4 previous similar messages [13641.314331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13641.322098] Lustre: Skipped 7 previous similar messages [13647.327769] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13647.458891] LustreError: 454179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13647.509708] Lustre: server umount lustre-MDT0000 complete [13651.379916] LustreError: 454161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798223 with bad export cookie 4767833845756231886 [13651.387910] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13651.404610] LustreError: 454161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13651.839512] Lustre: server umount lustre-MDT0001 complete [13670.943212] LNet: 1 local NIs in recovery (showing 1): 192.168.204.122@tcp [13685.347690] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [13697.667981] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [13740.914682] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:38:32 (1787798312) [13760.066205] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [13763.035326] Key type lgssc unregistered [13763.335733] LNet: 460798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13763.345220] LNetError: 460798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13870.895597] LNet: Removed LNI 192.168.204.122@tcp [13872.163284] Key type .llcrypt unregistered [13872.166734] Key type ._llcrypt unregistered [13890.093772] Key type ._llcrypt registered [13890.096700] Key type .llcrypt registered [13890.231580] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [13891.677874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13891.745568] alg: No test for adler32 (adler32-zlib) [13892.753119] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [13893.014494] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [13894.751210] Key type lgssc registered [13896.261006] Lustre: Echo OBD driver; http://www.lustre.org/ [13900.783630] LNet: Added LNI 192.168.204.122@tcp100 [8/256/0/180] [13901.666620] LNet: Added LNI 192.168.204.122@tcp101 [8/256/0/180] [13902.639259] LNet: Added LNI 192.168.204.122@tcp102 [8/256/0/180] [13903.521615] LNet: Added LNI 192.168.204.122@tcp103 [8/256/0/180] [13904.292848] LNet: Added LNI 192.168.204.122@tcp104 [8/256/0/180] [13905.103288] LNet: Added LNI 192.168.204.122@tcp105 [8/256/0/180] [13905.944236] LNet: Added LNI 192.168.204.122@tcp106 [8/256/0/180] [13906.610369] LNet: Added LNI 192.168.204.122@tcp107 [8/256/0/180] [13907.328589] LNet: Added LNI 192.168.204.122@tcp108 [8/256/0/180] [13908.282272] LNet: Added LNI 192.168.204.122@tcp109 [8/256/0/180] [13909.038984] LNet: Added LNI 192.168.204.122@tcp110 [8/256/0/180] [13909.886370] LNet: Added LNI 192.168.204.122@tcp111 [8/256/0/180] [13910.707922] LNet: Added LNI 192.168.204.122@tcp112 [8/256/0/180] [13911.788404] LNet: Added LNI 192.168.204.122@tcp113 [8/256/0/180] [13912.582278] LNet: Added LNI 192.168.204.122@tcp114 [8/256/0/180] [13913.351103] LNet: Added LNI 192.168.204.122@tcp115 [8/256/0/180] [13914.205890] LNet: Added LNI 192.168.204.122@tcp116 [8/256/0/180] [13915.171298] LNet: Added LNI 192.168.204.122@tcp117 [8/256/0/180] [13916.205994] LNet: Added LNI 192.168.204.122@tcp118 [8/256/0/180] [13917.412057] LNet: Added LNI 192.168.204.122@tcp119 [8/256/0/180] [13918.788770] LNet: Added LNI 192.168.204.122@tcp120 [8/256/0/180] [13919.770085] LNet: Added LNI 192.168.204.122@tcp121 [8/256/0/180] [13920.906145] LNet: Added LNI 192.168.204.122@tcp122 [8/256/0/180] [13922.072274] LNet: Added LNI 192.168.204.122@tcp123 [8/256/0/180] [13923.387550] LNet: Added LNI 192.168.204.122@tcp124 [8/256/0/180] [13924.403470] LNet: Added LNI 192.168.204.122@tcp125 [8/256/0/180] [13925.336616] LNet: Added LNI 192.168.204.122@tcp126 [8/256/0/180] [13926.253904] LNet: Added LNI 192.168.204.122@tcp127 [8/256/0/180] [13927.166924] LNet: Added LNI 192.168.204.122@tcp128 [8/256/0/180] [13928.278570] LNet: Added LNI 192.168.204.122@tcp129 [8/256/0/180] [13928.944278] LNet: Added LNI 192.168.204.122@tcp130 [8/256/0/180] [13929.789342] LNet: Added LNI 192.168.204.122@tcp131 [8/256/0/180] [13930.428115] LNet: Added LNI 192.168.204.122@tcp132 [8/256/0/180] [13931.400731] LNet: Added LNI 192.168.204.122@tcp133 [8/256/0/180] [13932.442177] LNet: Added LNI 192.168.204.122@tcp134 [8/256/0/180] [13933.287746] LNet: Added LNI 192.168.204.122@tcp135 [8/256/0/180] [13938.900864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13938.915510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13940.044707] Lustre: 463831:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.122@tcp: can't add NID 192.168.204.122@tcp130: rc = -75 [13940.131918] LNetError: 461290:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13940.150029] LNetError: Protocol error connecting to 192.168.204.122@tcp113 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13940.167778] LNetError: Skipped 6 previous similar messages [13940.262932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13940.286586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13940.339649] Lustre: lustre-MDT0000: new disk, initializing [13940.389349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13940.400812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13941.151316] LNet: 9 local NIs in recovery (showing 9): 192.168.204.122@tcp121, 192.168.204.122@tcp120, 192.168.204.122@tcp122, 192.168.204.122@tcp123, 192.168.204.122@tcp119, 192.168.204.122@tcp126, 192.168.204.122@tcp125, 192.168.204.122@tcp127, 192.168.204.122@tcp128 [13941.153184] LNetError: 463853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13941.180533] LNetError: 463853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [13941.188504] LNetError: Protocol error connecting to 192.168.204.122@tcp121 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13941.197328] LNetError: Skipped 6 previous similar messages [13943.203983] LNetError: 463851:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13943.214547] LNetError: 463851:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [13943.218205] LNetError: Protocol error connecting to 192.168.204.122@tcp121 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13943.222451] LNetError: Skipped 2 previous similar messages [13944.031574] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13947.302968] LNetError: 461289:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13947.310968] LNetError: Protocol error connecting to 192.168.204.122@tcp121 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13954.116034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13954.297384] Lustre: 463849:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13954.353669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13954.357893] Lustre: Skipped 1 previous similar message [13954.455420] Lustre: 464652:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.122@tcp: can't add NID 192.168.204.122@tcp130: rc = -75 [13954.466101] Lustre: 464652:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13954.501124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13954.529770] Lustre: lustre-MDT0001: new disk, initializing [13954.659777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13954.707578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13954.722596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13955.488530] LNetError: 461290:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13955.512052] LNetError: Protocol error connecting to 192.168.204.122@tcp121 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13959.558948] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13969.258825] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13970.917183] LNetError: 463931:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13970.930120] LNetError: Protocol error connecting to 192.168.204.122@tcp124 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13975.651954] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13983.185100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13983.629932] Lustre: lustre-OST0000: new disk, initializing [13983.633135] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13983.639329] Lustre: 465822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13983.741850] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.122@tcp: can't add NID 192.168.204.122@tcp130: rc = -75 [13983.746869] Lustre: 465827:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [13983.787498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13987.297966] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [13987.314283] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [13987.327444] LNetError: Protocol error connecting to 192.168.204.122@tcp117 at host 192.168.204.122:988: is it running a compatible version of Lustre? [13987.346949] LNetError: Skipped 9 previous similar messages [13989.887011] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [13992.506178] Lustre: 466278:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.122@tcp: can't add NID 192.168.204.122@tcp130: rc = -75 [13992.512875] Lustre: 466278:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13992.557540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13992.581636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13992.720765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13999.493464] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14001.567165] LNet: 1 local NIs in recovery (showing 1): 192.168.204.122@tcp121 [14008.305309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14008.306690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14008.313443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14008.324211] Lustre: Skipped 1 previous similar message [14008.334774] Lustre: Skipped 1 previous similar message [14013.497369] Lustre: server umount lustre-OST0000 complete [14018.530129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14018.542924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14018.553677] Lustre: Skipped 1 previous similar message [14023.344192] Lustre: server umount lustre-MDT0000 complete [14023.649076] LustreError: 463863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14023.665963] LustreError: 463863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14026.940641] LustreError: 463839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787798598 with bad export cookie 15365013459000394217 [14026.952937] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14026.972133] LustreError: 463839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14027.322834] Lustre: server umount lustre-MDT0001 complete [14034.401521] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.122 [14034.413579] LNetError: 461287:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [14034.425809] LNetError: Protocol error connecting to 192.168.204.122@tcp124 at host 192.168.204.122:988: is it running a compatible version of Lustre? [14034.438181] LNetError: Skipped 19 previous similar messages [14037.945412] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [14041.143468] Key type lgssc unregistered [14041.519795] LNet: 467739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14041.528197] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14041.537970] LNet: Removed LNI 192.168.204.122@tcp [14042.591197] LNet: Removed LNI 192.168.204.122@tcp100 [14042.598365] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14042.606114] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14042.612917] LNet: Removed LNI 192.168.204.122@tcp101 [14043.681884] LNet: Removed LNI 192.168.204.122@tcp102 [14043.694380] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14043.712702] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14043.719285] LNet: Removed LNI 192.168.204.122@tcp103 [14044.767825] LNet: Removed LNI 192.168.204.122@tcp104 [14045.791291] LNet: Removed LNI 192.168.204.122@tcp105 [14045.799768] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14045.814695] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14045.835259] LNet: Removed LNI 192.168.204.122@tcp106 [14046.879443] LNet: Removed LNI 192.168.204.122@tcp107 [14047.903558] LNet: Removed LNI 192.168.204.122@tcp108 [14048.927148] LNet: Removed LNI 192.168.204.122@tcp109 [14049.951154] LNet: Removed LNI 192.168.204.122@tcp110 [14049.980962] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14049.989050] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14050.004232] LNet: Removed LNI 192.168.204.122@tcp111 [14051.045166] LNet: Removed LNI 192.168.204.122@tcp112 [14052.066515] LNet: Removed LNI 192.168.204.122@tcp113 [14053.087107] LNet: Removed LNI 192.168.204.122@tcp114 [14054.111203] LNet: Removed LNI 192.168.204.122@tcp115 [14055.200217] LNet: Removed LNI 192.168.204.122@tcp116 [14056.224852] LNet: Removed LNI 192.168.204.122@tcp117 [14057.248290] LNet: Removed LNI 192.168.204.122@tcp118 [14058.271492] LNet: Removed LNI 192.168.204.122@tcp119 [14058.278365] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14058.291764] LNetError: 467739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14058.300078] LNet: Removed LNI 192.168.204.122@tcp120 [14058.307459] LNet: Removed LNI 192.168.204.122@tcp121 [14059.359169] LNet: Removed LNI 192.168.204.122@tcp122 [14060.383735] LNet: Removed LNI 192.168.204.122@tcp123 [14061.409877] LNet: Removed LNI 192.168.204.122@tcp124 [14062.431427] LNet: Removed LNI 192.168.204.122@tcp125 [14063.455591] LNet: Removed LNI 192.168.204.122@tcp126 [14064.479202] LNet: Removed LNI 192.168.204.122@tcp127 [14065.504920] LNet: Removed LNI 192.168.204.122@tcp128 [14066.528302] LNet: Removed LNI 192.168.204.122@tcp129 [14066.530125] LNet: Removed LNI 192.168.204.122@tcp130 [14066.533883] LNet: Removed LNI 192.168.204.122@tcp131 [14066.537321] LNet: Removed LNI 192.168.204.122@tcp132 [14066.540327] LNet: Removed LNI 192.168.204.122@tcp133 [14066.543196] LNet: Removed LNI 192.168.204.122@tcp134 [14066.552900] LNet: Removed LNI 192.168.204.122@tcp135 [14067.239777] Key type .llcrypt unregistered [14067.242597] Key type ._llcrypt unregistered [14088.763299] Key type ._llcrypt registered [14088.768290] Key type .llcrypt registered [14088.896776] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [14102.941656] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [14103.787207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14103.810376] alg: No test for adler32 (adler32-zlib) [14104.973833] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [14105.262537] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [14106.951229] Key type lgssc registered [14107.895368] Lustre: Echo OBD driver; http://www.lustre.org/ [14145.044920] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 22:45:16 (1787798716) [14163.533405] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [14166.338991] Key type lgssc unregistered [14166.794888] LNet: 471828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14166.810811] LNetError: 471828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14166.841584] LNet: Removed LNI 192.168.204.122@tcp [14167.706829] Key type .llcrypt unregistered [14167.712859] Key type ._llcrypt unregistered [14189.147607] Key type ._llcrypt registered [14189.159826] Key type .llcrypt registered [14189.344332] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [14190.731114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14191.101435] alg: No test for adler32 (adler32-zlib) [14192.219834] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [14192.717377] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [14194.495184] Key type lgssc registered [14195.577225] Lustre: Echo OBD driver; http://www.lustre.org/ [14207.524400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14207.560751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14208.998837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14209.083888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14209.213606] Lustre: lustre-MDT0000: new disk, initializing [14209.339076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14209.361601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14214.765268] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14228.255110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14228.443615] Lustre: 473043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14228.513500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14228.517671] Lustre: Skipped 1 previous similar message [14228.656995] Lustre: lustre-MDT0001: new disk, initializing [14228.761065] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14228.815440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14228.831463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14234.217601] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14243.551304] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14249.189382] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14255.536176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14256.018252] Lustre: lustre-OST0000: new disk, initializing [14256.036631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14256.056628] Lustre: 475013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14256.150756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14261.916775] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14263.863460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14263.876533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14263.956330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14271.083175] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14274.588460] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing lnet_if_list [14279.548302] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14279.605393] LNet: Added LNI 192.168.204.122@tcp10 [8/256/0/180] [14285.716740] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14285.778414] LNet: Added LNI 192.168.204.122@tcp11 [8/256/0/180] [14289.968855] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14290.010219] LNet: Added LNI 192.168.204.122@tcp12 [8/256/0/180] [14294.467517] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14294.514742] LNet: Added LNI 192.168.204.122@tcp13 [8/256/0/180] [14299.126805] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14299.149982] LNet: Added LNI 192.168.204.122@tcp14 [8/256/0/180] [14303.573276] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14303.609811] LNet: Added LNI 192.168.204.122@tcp15 [8/256/0/180] [14309.271902] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14309.345132] LNet: Added LNI 192.168.204.122@tcp16 [8/256/0/180] [14317.474711] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14317.519642] LNet: Added LNI 192.168.204.122@tcp17 [8/256/0/180] [14324.974438] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14325.052287] LNet: Added LNI 192.168.204.122@tcp18 [8/256/0/180] [14332.108470] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14332.158340] LNet: Added LNI 192.168.204.122@tcp19 [8/256/0/180] [14339.001946] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14339.053684] LNet: Added LNI 192.168.204.122@tcp20 [8/256/0/180] [14345.327267] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14345.363446] LNet: Added LNI 192.168.204.122@tcp21 [8/256/0/180] [14352.439966] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14352.478870] LNet: Added LNI 192.168.204.122@tcp22 [8/256/0/180] [14360.130157] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14360.168636] LNet: Added LNI 192.168.204.122@tcp23 [8/256/0/180] [14367.007690] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14367.056262] LNet: Added LNI 192.168.204.122@tcp24 [8/256/0/180] [14373.639707] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14373.696402] LNet: Added LNI 192.168.204.122@tcp25 [8/256/0/180] [14381.162235] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14381.192269] LNet: Added LNI 192.168.204.122@tcp26 [8/256/0/180] [14387.811141] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14387.868177] LNet: Added LNI 192.168.204.122@tcp27 [8/256/0/180] [14395.743616] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14395.798873] LNet: Added LNI 192.168.204.122@tcp28 [8/256/0/180] [14403.725654] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14403.784633] LNet: Added LNI 192.168.204.122@tcp29 [8/256/0/180] [14411.149152] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14411.196977] LNet: Added LNI 192.168.204.122@tcp30 [8/256/0/180] [14419.602275] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14419.638681] LNet: Added LNI 192.168.204.122@tcp31 [8/256/0/180] [14427.310479] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14427.336994] LNet: Added LNI 192.168.204.122@tcp32 [8/256/0/180] [14434.449084] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14434.481831] LNet: Added LNI 192.168.204.122@tcp33 [8/256/0/180] [14441.427670] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14441.494637] LNet: Added LNI 192.168.204.122@tcp34 [8/256/0/180] [14449.115513] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14449.143884] LNet: Added LNI 192.168.204.122@tcp35 [8/256/0/180] [14456.758523] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14456.795296] LNet: Added LNI 192.168.204.122@tcp36 [8/256/0/180] [14464.315987] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14464.348182] LNet: Added LNI 192.168.204.122@tcp37 [8/256/0/180] [14471.878249] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14471.917921] LNet: Added LNI 192.168.204.122@tcp38 [8/256/0/180] [14479.806135] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14479.832996] LNet: Added LNI 192.168.204.122@tcp39 [8/256/0/180] [14488.515594] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14488.556839] LNet: Added LNI 192.168.204.122@tcp40 [8/256/0/180] [14496.958141] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14496.999269] LNet: Added LNI 192.168.204.122@tcp41 [8/256/0/180] [14504.992674] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14505.054044] LNet: Added LNI 192.168.204.122@tcp42 [8/256/0/180] [14513.490760] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14513.525738] LNet: Added LNI 192.168.204.122@tcp43 [8/256/0/180] [14522.314781] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14522.372926] LNet: Added LNI 192.168.204.122@tcp44 [8/256/0/180] [14534.087273] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14534.176941] LNet: Added LNI 192.168.204.122@tcp45 [8/256/0/180] [14546.950850] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14546.986056] LNet: Added LNI 192.168.204.122@tcp46 [8/256/0/180] [14556.898984] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14556.928126] LNet: Added LNI 192.168.204.122@tcp47 [8/256/0/180] [14566.401191] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14566.441206] LNet: Added LNI 192.168.204.122@tcp48 [8/256/0/180] [14575.673104] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14575.719745] LNet: Added LNI 192.168.204.122@tcp49 [8/256/0/180] [14586.522812] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14586.567305] LNet: Added LNI 192.168.204.122@tcp50 [8/256/0/180] [14607.330441] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14607.335732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14607.342487] Lustre: Skipped 1 previous similar message [14607.352426] Lustre: Skipped 2 previous similar messages [14612.330349] Lustre: server umount lustre-MDT0000 complete [14616.054208] LustreError: 473034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799188 with bad export cookie 13976397300517821476 [14616.054691] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14616.064981] LustreError: 473034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14616.401769] Lustre: server umount lustre-MDT0001 complete [14633.887448] Lustre: 472314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799189/real 1787799189] req@00000000c141e58c x1874642876944000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787799205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14633.916657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14633.928499] Lustre: Skipped 1 previous similar message [14634.465222] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [14637.023790] Lustre: 472317:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799193/real 1787799193] req@0000000015d46bdd x1874642876944128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787799209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14638.048773] Lustre: 472316:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799194/real 1787799194] req@000000008093c2a6 x1874642876944256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787799210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14642.091848] Lustre: 472314:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799198/real 1787799198] req@00000000d99d706b x1874642876944512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787799214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14642.866401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14647.263669] Lustre: 472315:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787799203/real 1787799203] req@00000000473ff3bb x1874642876944640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787799219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14661.600156] LustreError: 472313:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006bcda078 x1874642876945408/t0(0) o250->MGC192.168.204.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14661.996434] LustreError: 475005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14662.019440] LustreError: 475005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14662.041311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14663.073827] LustreError: 475006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14663.084520] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14663.089276] LustreError: 475006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14666.375386] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14668.258532] LustreError: 475005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.383347] LustreError: 475006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14673.681588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14674.021951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14677.903323] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14678.128109] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14684.091234] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14689.010438] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14709.216701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14709.222616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14709.237610] Lustre: Skipped 1 previous similar message [14709.256136] Lustre: Skipped 1 previous similar message [14714.342859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14714.347181] Lustre: Skipped 1 previous similar message [14714.984911] Lustre: server umount lustre-OST0000 complete [14719.457318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14719.462677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14719.486656] Lustre: Skipped 1 previous similar message [14719.494784] Lustre: Skipped 1 previous similar message [14724.577116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14724.584642] Lustre: Skipped 1 previous similar message [14725.517972] Lustre: server umount lustre-MDT0000 complete [14728.921804] LustreError: 473036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799300 with bad export cookie 13976397300517824199 [14728.932122] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14728.933839] LustreError: 473036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14729.183849] Lustre: server umount lustre-MDT0001 complete [14739.670862] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [14743.034424] Key type lgssc unregistered [14743.440358] LNet: 502066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14743.447956] LNetError: 502066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14744.479397] LNet: Removed LNI 192.168.204.122@tcp [14744.482726] LNetError: 502066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14744.493808] LNet: Removed LNI 192.168.204.122@tcp10 [14744.501234] LNet: Removed LNI 192.168.204.122@tcp11 [14744.503319] LNet: Removed LNI 192.168.204.122@tcp12 [14744.509860] LNet: Removed LNI 192.168.204.122@tcp13 [14744.516441] LNet: Removed LNI 192.168.204.122@tcp14 [14744.519841] LNet: Removed LNI 192.168.204.122@tcp15 [14744.525876] LNet: Removed LNI 192.168.204.122@tcp16 [14744.532766] LNet: Removed LNI 192.168.204.122@tcp17 [14744.535779] LNet: Removed LNI 192.168.204.122@tcp18 [14744.537716] LNet: Removed LNI 192.168.204.122@tcp19 [14744.547766] LNet: Removed LNI 192.168.204.122@tcp20 [14744.551720] LNet: Removed LNI 192.168.204.122@tcp21 [14744.563311] LNet: Removed LNI 192.168.204.122@tcp22 [14744.569168] LNet: Removed LNI 192.168.204.122@tcp23 [14744.580201] LNet: Removed LNI 192.168.204.122@tcp24 [14744.582280] LNet: Removed LNI 192.168.204.122@tcp25 [14744.590644] LNet: Removed LNI 192.168.204.122@tcp26 [14744.593827] LNet: Removed LNI 192.168.204.122@tcp27 [14744.596722] LNet: Removed LNI 192.168.204.122@tcp28 [14744.602190] LNet: Removed LNI 192.168.204.122@tcp29 [14744.605507] LNet: Removed LNI 192.168.204.122@tcp30 [14744.608422] LNet: Removed LNI 192.168.204.122@tcp31 [14744.612250] LNet: Removed LNI 192.168.204.122@tcp32 [14744.618264] LNet: Removed LNI 192.168.204.122@tcp33 [14744.625332] LNet: Removed LNI 192.168.204.122@tcp34 [14744.630238] LNet: Removed LNI 192.168.204.122@tcp35 [14744.632469] LNet: Removed LNI 192.168.204.122@tcp36 [14744.637164] LNet: Removed LNI 192.168.204.122@tcp37 [14744.641411] LNet: Removed LNI 192.168.204.122@tcp38 [14744.644941] LNet: Removed LNI 192.168.204.122@tcp39 [14744.657243] LNet: Removed LNI 192.168.204.122@tcp40 [14744.665231] LNet: Removed LNI 192.168.204.122@tcp41 [14744.674702] LNet: Removed LNI 192.168.204.122@tcp42 [14744.676880] LNet: Removed LNI 192.168.204.122@tcp43 [14744.684801] LNet: Removed LNI 192.168.204.122@tcp44 [14744.691543] LNet: Removed LNI 192.168.204.122@tcp45 [14744.695593] LNet: Removed LNI 192.168.204.122@tcp46 [14744.698834] LNet: Removed LNI 192.168.204.122@tcp47 [14744.701831] LNet: Removed LNI 192.168.204.122@tcp48 [14744.705145] LNet: Removed LNI 192.168.204.122@tcp49 [14744.716977] LNet: Removed LNI 192.168.204.122@tcp50 [14745.400764] Key type .llcrypt unregistered [14745.403613] Key type ._llcrypt unregistered [14770.934376] Key type ._llcrypt registered [14770.938205] Key type .llcrypt registered [14771.088262] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [14790.102670] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [14791.308752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14791.350483] alg: No test for adler32 (adler32-zlib) [14792.635463] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [14793.202350] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [14794.927330] Key type lgssc registered [14796.362975] Lustre: Echo OBD driver; http://www.lustre.org/ [14840.485491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14840.520710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14841.958701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14842.034268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14842.131430] Lustre: lustre-MDT0000: new disk, initializing [14842.272929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14842.310293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14847.760610] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14860.010956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14860.176632] Lustre: 505266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14860.244324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14860.249719] Lustre: Skipped 1 previous similar message [14860.345744] Lustre: lustre-MDT0001: new disk, initializing [14860.486231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14860.531929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14860.560652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14867.059824] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14876.989662] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14882.476254] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14889.767304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14889.965319] Lustre: *** cfs_fail_loc=90f, val=0*** [14890.052280] Lustre: lustre-OST0000: new disk, initializing [14890.055592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14890.064329] Lustre: 507280:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14890.167534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14895.511347] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [14896.187644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14896.197277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14896.284655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14905.059902] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14917.091994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14917.101379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14917.109121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14922.214752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14922.224830] Lustre: Skipped 2 previous similar messages [14927.327551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14927.336825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14927.341709] Lustre: Skipped 1 previous similar message [14927.633259] Lustre: server umount lustre-OST0000 complete [14932.457077] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14932.465894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14932.481327] Lustre: Skipped 2 previous similar messages [14932.484024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14932.494703] Lustre: Skipped 1 previous similar message [14932.525598] LustreError: Skipped 1 previous similar message [14937.569499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14937.579679] Lustre: Skipped 1 previous similar message [14946.271166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14946.652364] Lustre: server umount lustre-MDT0000 complete [14950.870910] LustreError: 505257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799522 with bad export cookie 10000139504334591343 [14950.880942] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14950.891495] LustreError: 505257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14951.487783] Lustre: server umount lustre-MDT0001 complete [14963.518329] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [14966.682731] Key type lgssc unregistered [14966.955599] LNet: 509192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14966.963677] LNetError: 509192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14966.982158] LNet: Removed LNI 192.168.204.122@tcp [14967.932151] Key type .llcrypt unregistered [14967.934532] Key type ._llcrypt unregistered [14992.434488] Key type ._llcrypt registered [14992.437501] Key type .llcrypt registered [14992.559136] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [14993.985733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14994.042435] alg: No test for adler32 (adler32-zlib) [14995.218681] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [14995.456253] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [14997.143507] Key type lgssc registered [14998.475710] Lustre: Echo OBD driver; http://www.lustre.org/ [15007.371578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15007.387404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15008.887658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15013.345816] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15018.470850] LustreError: 510469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15018.491282] LustreError: 510469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15021.806469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15022.226497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15027.944979] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15036.252782] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15042.674441] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15049.892456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15050.222578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15056.724245] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15065.195524] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15076.568600] loop0: detected capacity change from 0 to 204800000 [15076.735478] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15076.764885] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15076.775560] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15083.205755] loop0: detected capacity change from 0 to 204800000 [15083.415254] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15083.426489] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15083.436109] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15089.113652] loop0: detected capacity change from 0 to 204800000 [15094.462363] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15094.607436] Lustre: fs15246-OST0000: new disk, initializing [15094.609603] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15094.613772] Lustre: 514076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15094.682230] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15101.454790] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15110.979751] loop1: detected capacity change from 0 to 204800000 [15116.535734] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15116.624282] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15116.684887] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15116.789237] Lustre: fs15246-MDT0000: new disk, initializing [15117.002462] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15117.037790] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15117.192325] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15118.241797] Lustre: 515200:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000080c5a050 x1874643718813952/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787799706 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15121.344366] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15127.336921] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15127.353175] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15127.381615] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15144.426879] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15144.446871] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15149.545291] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15149.848328] Lustre: server umount fs15246-MDT0000 complete [15154.673849] LustreError: 510463:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15159.785961] LustreError: 510464:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15160.635359] Lustre: server umount fs15246-OST0000 complete [15175.138836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15175.153745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15175.157633] Lustre: Skipped 1 previous similar message [15175.178275] Lustre: Skipped 1 previous similar message [15180.257755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15180.275969] Lustre: Skipped 1 previous similar message [15180.716985] Lustre: server umount lustre-OST0000 complete [15190.500695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15190.513615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15190.518938] Lustre: Skipped 1 previous similar message [15192.317069] Lustre: server umount lustre-MDT0000 complete [15195.619956] LustreError: 510468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15195.649328] LustreError: 510468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15196.726093] LustreError: 510449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787799768 with bad export cookie 5217612447866359102 [15196.736421] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15196.749093] LustreError: 510449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15197.347499] Lustre: server umount lustre-MDT0001 complete [15208.854220] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [15212.169558] Key type lgssc unregistered [15212.520037] LNet: 518298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15212.531603] LNetError: 518298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15212.548706] LNet: Removed LNI 192.168.204.122@tcp [15213.703116] Key type .llcrypt unregistered [15213.706472] Key type ._llcrypt unregistered [15232.902680] Key type ._llcrypt registered [15232.905698] Key type .llcrypt registered [15233.051623] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [15234.108346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15234.123685] alg: No test for adler32 (adler32-zlib) [15235.244620] Lustre: Lustre: Build Version: 2.17.57_54_g0142cbb [15235.497252] LNet: Added LNI 192.168.204.122@tcp [8/256/0/180] [15237.175283] Key type lgssc registered [15238.354180] Lustre: Echo OBD driver; http://www.lustre.org/ [15283.143953] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [15295.611719] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [15338.898811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15338.933373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15340.221626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15340.276447] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15340.350304] Lustre: lustre-MDT0000: new disk, initializing [15340.461238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15340.501881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15345.048426] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15356.450707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15356.584154] Lustre: 522769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15356.634544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15356.637787] Lustre: Skipped 1 previous similar message [15356.691371] Lustre: lustre-MDT0001: new disk, initializing [15356.773858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15356.809695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15356.821499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15361.139456] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15370.851962] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15377.433584] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15384.705761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15385.118353] Lustre: lustre-OST0000: new disk, initializing [15385.122254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15385.131448] Lustre: 524742:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15385.208549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15391.597857] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_default_debug -1 all [15392.880149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15392.887309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15393.000448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15402.030538] Lustre: DEBUG MARKER: oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15407.396722] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15407.746137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15412.293681] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15412.483661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15418.856031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15418.856885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15418.859758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15418.872162] LustreError: Skipped 1 previous similar message [15418.881554] Lustre: Skipped 1 previous similar message [15420.409121] Lustre: server umount lustre-OST0000 complete [15428.576654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15428.584133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15428.596438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15428.602787] Lustre: Skipped 1 previous similar message [15430.492954] Lustre: server umount lustre-MDT0000 complete [15434.381747] LustreError: 523567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787800006 with bad export cookie 7037849688254905875 [15434.387838] LustreError: MGC192.168.204.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15434.395467] LustreError: 523567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15434.744441] Lustre: server umount lustre-MDT0001 complete [15460.721777] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing set_hostid [15472.150712] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing load_modules_local [15505.726689] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15240 sec ========= 23:07:57 (1787800077) [15507.487469] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:07:59 (1787800079) === [15510.829248] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:08:02 (1787800082) === [15552.322501] Lustre: DEBUG MARKER: oleg422-server.virtnet: executing unload_modules_local [15556.282651] Key type lgssc unregistered [15556.683093] LNet: 531388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15556.701287] LNetError: 531388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15556.730280] LNet: Removed LNI 192.168.204.122@tcp [15557.575578] Key type .llcrypt unregistered [15557.580130] Key type ._llcrypt unregistered