[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468609326 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002350] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007039] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010057] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013100] *** VALIDATE selinux *** [ 0.021533] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026686] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] *** VALIDATE tmpfs *** [ 0.030484] *** VALIDATE proc *** [ 0.032251] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041488] debug: unmapping init [mem 0xffffffffb1259000-0xffffffffb1260fff] [ 0.043181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046018] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048016] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050016] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052342] rcu: Hierarchical SRCU implementation. [ 0.054547] smp: Bringing up secondary CPUs ... [ 0.055649] x86: Booting SMP configuration: [ 0.056032] .... node #0, CPUs: #1 #2 #3 [ 0.060106] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.177551] node 0 deferred pages initialised in 111ms [ 0.181190] devtmpfs: initialized [ 0.182338] x86/mm: Memory block size: 128MB [ 0.185485] gcov: version magic: 0x41383552 [ 0.188294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.195304] pinctrl core: initialized pinctrl subsystem [ 0.197160] [ 0.197668] ************************************************************* [ 0.200013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203013] ** ** [ 0.205011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.207014] ** ** [ 0.209012] ** This means that this kernel is built to expose internal ** [ 0.211011] ** IOMMU data structures, which may compromise security on ** [ 0.213009] ** your system. ** [ 0.217014] ** ** [ 0.219011] ** If you see this message and you are not debugging the ** [ 0.221012] ** kernel, report this immediately to your vendor! ** [ 0.224014] ** ** [ 0.226016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229015] ************************************************************* [ 0.232760] NET: Registered protocol family 16 [ 0.234538] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.237068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.240085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.244163] cpuidle: using governor menu [ 0.246542] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.250884] PCI: Using configuration type 1 for base access [ 0.253164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.262153] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.263042] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.265094] cryptd: max_cpu_qlen set to 1000 [ 0.267302] ACPI: Added _OSI(Module Device) [ 0.269016] ACPI: Added _OSI(Processor Device) [ 0.270069] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.272013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278500] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.285682] ACPI: Interpreter enabled [ 0.287139] ACPI: PM: (supports S0 S3 S4 S5) [ 0.289017] ACPI: Using IOAPIC for interrupt routing [ 0.291161] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295448] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.304966] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.307038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.309023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.313087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.318381] acpiphp: Slot [2] registered [ 0.320131] acpiphp: Slot [5] registered [ 0.321136] acpiphp: Slot [6] registered [ 0.323121] acpiphp: Slot [7] registered [ 0.325141] acpiphp: Slot [8] registered [ 0.326084] acpiphp: Slot [9] registered [ 0.326920] acpiphp: Slot [10] registered [ 0.328146] acpiphp: Slot [3] registered [ 0.330118] acpiphp: Slot [4] registered [ 0.331118] acpiphp: Slot [11] registered [ 0.333120] acpiphp: Slot [12] registered [ 0.334104] acpiphp: Slot [13] registered [ 0.335167] acpiphp: Slot [14] registered [ 0.337110] acpiphp: Slot [15] registered [ 0.339132] acpiphp: Slot [16] registered [ 0.341103] acpiphp: Slot [17] registered [ 0.342108] acpiphp: Slot [18] registered [ 0.344142] acpiphp: Slot [19] registered [ 0.346116] acpiphp: Slot [20] registered [ 0.347104] acpiphp: Slot [21] registered [ 0.349174] acpiphp: Slot [22] registered [ 0.351139] acpiphp: Slot [23] registered [ 0.353114] acpiphp: Slot [24] registered [ 0.354000] acpiphp: Slot [25] registered [ 0.357125] acpiphp: Slot [26] registered [ 0.358104] acpiphp: Slot [27] registered [ 0.360147] acpiphp: Slot [28] registered [ 0.362110] acpiphp: Slot [29] registered [ 0.363121] acpiphp: Slot [30] registered [ 0.365126] acpiphp: Slot [31] registered [ 0.367067] PCI host bridge to bus 0000:00 [ 0.368017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.371026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.373023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.379029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.382024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.384183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388101] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.391311] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.401030] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.406057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.408024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.414021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.417046] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.419896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.422121] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.425850] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.432016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.446022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.451023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456966] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.464017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.469012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.484024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.497387] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.503017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.512070] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.543018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561387] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.575016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.596020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606854] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.614019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.623018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.657587] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.669019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711050] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.725014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.740019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.754633] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.756357] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.760432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.763420] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.765243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.769122] iommu: Default domain type: Passthrough [ 0.771615] SCSI subsystem initialized [ 0.774236] ACPI: bus type USB registered [ 0.775180] usbcore: registered new interface driver usbfs [ 0.778156] usbcore: registered new interface driver hub [ 0.780195] usbcore: registered new device driver usb [ 0.782319] pps_core: LinuxPPS API ver. 1 registered [ 0.784020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787119] PTP clock support registered [ 0.790088] EDAC MC: Ver: 3.0.0 [ 0.793321] PCI: Using ACPI for IRQ routing [ 0.794815] NetLabel: Initializing [ 0.796027] NetLabel: domain hash size = 128 [ 0.798019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.800128] NetLabel: unlabeled traffic allowed by default [ 0.803153] vgaarb: loaded [ 0.804426] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.806016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.812474] clocksource: Switched to clocksource kvm-clock [ 0.920859] VFS: Disk quotas dquot_6.6.0 [ 0.923335] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.926165] *** VALIDATE ramfs *** [ 0.927809] *** VALIDATE hugetlbfs *** [ 0.929583] pnp: PnP ACPI init [ 0.932113] pnp: PnP ACPI: found 6 devices [ 0.954044] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.957611] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.959907] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.962021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.964837] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.967276] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.970186] NET: Registered protocol family 2 [ 0.972667] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.977509] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.981181] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.986730] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.989795] TCP: Hash tables configured (established 65536 bind 65536) [ 0.992904] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.996294] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.999190] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.002501] NET: Registered protocol family 1 [ 1.006508] RPC: Registered named UNIX socket transport module. [ 1.008818] RPC: Registered udp transport module. [ 1.010699] RPC: Registered tcp transport module. [ 1.012659] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.014995] NET: Registered protocol family 44 [ 1.016790] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.019052] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021339] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.023738] PCI: CLS 0 bytes, default 64 [ 1.024877] Unpacking initramfs... [ 2.437549] debug: unmapping init [mem 0xffff9c1f3cc54000-0xffff9c1f3ffbffff] [ 2.441804] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444174] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447523] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.953766] Initialise system trusted keyrings [ 2.955695] Key type blacklist registered [ 2.957493] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.971464] zbud: loaded [ 2.974780] *** VALIDATE nfs *** [ 2.976134] *** VALIDATE nfs4 *** [ 2.977850] pstore: using deflate compression [ 2.982033] Platform Keyring initialized [ 3.118324] NET: Registered protocol family 38 [ 3.120535] Key type asymmetric registered [ 3.122143] Asymmetric key parser 'x509' registered [ 3.124375] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.127457] io scheduler mq-deadline registered [ 3.129266] io scheduler kyber registered [ 3.131018] io scheduler bfq registered [ 3.133130] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.136349] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.139581] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.143096] ACPI: Power Button [PWRF] [ 3.148792] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.156044] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.172173] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.180722] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.204959] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233307] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.262058] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.267994] Non-volatile memory driver v1.3 [ 3.269983] Linux agpgart interface v0.103 [ 3.306840] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.311403] vda: detected capacity change from 0 to 76677120 [ 3.332407] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.335283] vdb: detected capacity change from 0 to 1073741824 [ 3.352555] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.355828] vdc: detected capacity change from 0 to 2621440000 [ 3.375428] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.378909] vdd: detected capacity change from 0 to 2621440000 [ 3.394878] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.398367] vde: detected capacity change from 0 to 4294967296 [ 3.413210] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416575] vdf: detected capacity change from 0 to 4294967296 [ 3.424159] libphy: Fixed MDIO Bus: probed [ 3.430505] usbcore: registered new interface driver usbserial_generic [ 3.433270] usbserial: USB Serial support registered for generic [ 3.435876] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.439523] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.441086] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.443934] mousedev: PS/2 mouse device common for all mice [ 3.447412] rtc_cmos 00:05: RTC can wake from S4 [ 3.450425] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.454353] rtc_cmos 00:05: registered as rtc0 [ 3.456186] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.458410] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459292] intel_pstate: CPU model not supported [ 3.463443] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.466561] hid: raw HID events driver (C) Jiri Kosina [ 3.468399] usbcore: registered new interface driver usbhid [ 3.470270] usbhid: USB HID core driver [ 3.471836] drop_monitor: Initializing network drop monitor service [ 3.474374] Initializing XFRM netlink socket [ 3.476428] NET: Registered protocol family 10 [ 3.479442] Segment Routing with IPv6 [ 3.481028] NET: Registered protocol family 17 [ 3.483376] mpls_gso: MPLS GSO support [ 3.489230] RAS: Correctable Errors collector initialized. [ 3.491477] AVX version of gcm_enc/dec engaged. [ 3.492986] AES CTR mode by8 optimization enabled [ 3.581118] sched_clock: Marking stable (3581026048, 0)->(4503635752, -922609704) [ 3.584546] registered taskstats version 1 [ 3.587140] Loading compiled-in X.509 certificates [ 3.589987] zswap: loaded using pool lzo/zbud [ 3.618154] Key type big_key registered [ 3.633130] Key type encrypted registered [ 3.634686] ima: No TPM chip found, activating TPM-bypass! [ 3.636791] ima: Allocated hash algorithm: sha1 [ 3.638518] ima: No architecture policies found [ 3.640237] evm: Initialising EVM extended attributes: [ 3.642161] evm: security.selinux [ 3.643462] evm: security.ima [ 3.644617] evm: security.capability [ 3.645948] evm: HMAC attrs: 0x1 [ 3.648418] rtc_cmos 00:05: setting system clock to 2026-09-08 13:29:24 UTC (1788874164) [ 3.655362] debug: unmapping init [mem 0xffffffffb2203000-0xffffffffb23fffff] [ 3.658650] debug: unmapping init [mem 0xffffffffb0f82000-0xffffffffb1258fff] [ 3.667076] Write protecting the kernel read-only data: 28672k [ 3.670646] debug: unmapping init [mem 0xffffffffaf603000-0xffffffffaf7fffff] [ 3.672812] debug: unmapping init [mem 0xffffffffaff14000-0xffffffffafffffff] [ 3.708124] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.717458] systemd[1]: Detected virtualization kvm. [ 3.719067] systemd[1]: Detected architecture x86-64. [ 3.721181] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.749343] systemd[1]: No hostname configured. [ 3.751067] systemd[1]: Set hostname to . [ 3.753110] random: systemd: uninitialized urandom read (16 bytes read) [ 3.755540] systemd[1]: Initializing machine ID from random generator. [ 3.789761] random: ln: uninitialized urandom read (6 bytes read) [ 3.870431] random: systemd: uninitialized urandom read (16 bytes read) [ 3.872966] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.879912] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.893705] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.512070] device-mapper: uevent: version 1.0.3 [ 4.514929] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.283791] virtio_net virtio0 ens2: renamed from eth0 [ 5.304035] random: fast init done [ 5.353266] scsi host0: ata_piix [ 5.366503] scsi host1: ata_piix [ 5.367993] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.370598] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.829852] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.151779] random: crng init done [ 10.153027] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.639613] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.919937] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.200917] SELinux: Disabled at runtime. [ 12.263319] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.273145] systemd[1]: Detected virtualization kvm. [ 12.275086] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.787168] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.790161] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.800929] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.805184] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.807464] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.813743] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.817706] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ 12.994859] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.245662] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.592845] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.654517] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.756794] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.775354] EDAC sbridge: Ver: 1.1.2 [ 15.209786] Key type dns_resolver registered [ 15.512849] NFS: Registering the id_resolver key type [ 15.517615] Key type id_resolver registered [ 15.519461] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-server login: [ 41.735420] libcfs: loading out-of-tree module taints kernel. [ 41.755285] Key type ._llcrypt registered [ 41.756867] Key type .llcrypt registered [ 41.806259] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 60.024562] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 61.523525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.548418] alg: No test for adler32 (adler32-zlib) [ 63.077849] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 64.147406] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 65.912657] Key type lgssc registered [ 68.303537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.505430] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 101.984693] hrtimer: interrupt took 16048143 ns [ 133.664277] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 147.897995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.935147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.235221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 149.286840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 149.405800] Lustre: lustre-MDT0000: new disk, initializing [ 149.495795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.515664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.083782] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 168.951445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.058886] Lustre: 6514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.110724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.121271] Lustre: Skipped 1 previous similar message [ 169.289924] Lustre: lustre-MDT0001: new disk, initializing [ 169.375637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.414554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 169.425385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 174.445856] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.392110] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.048858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.347268] Lustre: lustre-OST0000: new disk, initializing [ 189.350918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.359754] Lustre: 8450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.428609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.495838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.504917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.564979] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.578028] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.060242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.172178] Lustre: lustre-OST0001: new disk, initializing [ 210.181480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 210.188843] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.263206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.635329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.641248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.696162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.831281] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.180732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.345124] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.243573] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.562315] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing yml_node [ 262.639182] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 265.370473] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 267.775872] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 270.127749] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 09:33:47 EDT 2026 [ 293.993657] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 295.916141] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.801333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.812344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.832966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.826034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.827368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.220313] Lustre: server umount lustre-MDT0000 complete [ 324.077399] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.099716] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 324.276927] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874485 with bad export cookie 18290508191431406808 [ 324.284484] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 324.294720] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 329.185369] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.187377] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.227577] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 329.240977] Lustre: Skipped 2 previous similar messages [ 329.264545] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.271328] Lustre: Skipped 4 previous similar messages [ 330.862084] Lustre: server umount lustre-MDT0001 complete [ 336.291876] Lustre: server umount lustre-OST0000 complete [ 341.981911] Lustre: server umount lustre-OST0001 complete [ 350.940549] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 361.442582] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 402.450226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.670682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.701837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.780900] Lustre: lustre-MDT0000: new disk, initializing [ 402.908568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.939688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.828660] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 419.537905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.639210] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.709183] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.719942] Lustre: Skipped 1 previous similar message [ 419.852799] Lustre: lustre-MDT0001: new disk, initializing [ 420.038190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.077203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.108417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.445968] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 435.031984] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 441.381799] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.495823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.752832] Lustre: lustre-OST0000: new disk, initializing [ 448.756190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.762194] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.862706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.682068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.695088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.837176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.158351] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 465.919221] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 471.695135] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.966241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 476.908585] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 477.095471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.765986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.768356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.773602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.782890] Lustre: Skipped 2 previous similar messages [ 481.791724] Lustre: Skipped 1 previous similar message [ 485.129154] Lustre: server umount lustre-OST0000 complete [ 492.011067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.015494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.016502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.016520] LustreError: Skipped 1 previous similar message [ 492.021481] Lustre: Skipped 1 previous similar message [ 494.840456] Lustre: server umount lustre-MDT0000 complete [ 497.123290] LustreError: 15094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.149502] LustreError: 15094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 499.110315] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874659 with bad export cookie 18290508191431408341 [ 499.131736] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.573745] Lustre: server umount lustre-MDT0001 complete [ 504.746913] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:37:43 (1788874663) [ 507.771193] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 510.084842] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:37:48 (1788874668) [ 517.365080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.754530] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.824985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.618538] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 523.247478] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.761905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.070067] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 543.711970] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 550.565691] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 558.210120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 565.750597] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 573.997103] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 586.218781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.219097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 586.248948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.285796] Lustre: Skipped 2 previous similar messages [ 590.728709] Lustre: server umount lustre-OST0000 complete [ 600.420430] Lustre: server umount lustre-MDT0000 complete [ 601.569811] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.569811] LustreError: 19457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.569822] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 601.582560] LustreError: 19457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 603.946296] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874764 with bad export cookie 18290508191431409216 [ 603.952772] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.954858] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 604.342045] Lustre: server umount lustre-MDT0001 complete [ 614.940564] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 618.045122] Key type lgssc unregistered [ 618.416971] LNet: 22993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.423547] LNetError: 22993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.438894] LNet: Removed LNI 192.168.204.133@tcp [ 619.475179] Key type .llcrypt unregistered [ 619.479149] Key type ._llcrypt unregistered [ 639.491245] Key type ._llcrypt registered [ 639.496200] Key type .llcrypt registered [ 639.637865] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 640.408940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.515580] alg: No test for adler32 (adler32-zlib) [ 641.584904] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 641.955695] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 643.664171] Key type lgssc registered [ 644.644713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.671708] loop: module loaded [ 652.763755] loop0: detected capacity change from 0 to 204800000 [ 652.956042] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.966644] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 653.195775] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.332351] loop0: detected capacity change from 0 to 204800000 [ 659.541730] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.555663] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 659.573095] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 664.495958] loop0: detected capacity change from 0 to 204800000 [ 669.735393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 669.766892] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.094438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 671.138575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 671.227370] Lustre: lustre-MDT0000: new disk, initializing [ 671.326392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 671.352474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 676.233826] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 685.291637] loop1: detected capacity change from 0 to 204800000 [ 691.169678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.478585] Lustre: lustre-OST07c6: new disk, initializing [ 691.483272] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 691.492080] Lustre: Skipped 1 previous similar message [ 691.505477] Lustre: 26157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 691.588120] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 697.841288] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 700.517329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 700.546053] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 700.593791] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 702.355119] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 702.364088] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 702.376717] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 702.383827] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 702.391525] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 702.395625] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 720.067815] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 720.081356] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.090035] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.095894] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 720.103646] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.108993] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.051765] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 725.060349] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.066697] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 725.076328] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.081714] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.088617] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.100820] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 725.113666] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.123611] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.128796] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.134080] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.140810] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.332029] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 728.338608] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 728.345951] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 728.359755] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.373880] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.379897] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.385893] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 728.390713] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.394129] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.399677] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.407585] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.414902] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.734273] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 732.742897] Lustre: 25143:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 732.750804] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 732.760356] Lustre: 25143:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.770695] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 732.775556] Lustre: 25143:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.780148] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 732.790403] Lustre: 25143:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.798485] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 732.802653] Lustre: 25143:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.808283] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.812653] Lustre: 25143:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 743.617195] Lustre: 26165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 743.624252] Lustre: 26165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 743.629972] Lustre: 26165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 743.635683] Lustre: 26165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.640725] Lustre: 26165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 743.647869] Lustre: 26165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.653420] Lustre: 26165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 743.659535] Lustre: 26165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.664991] Lustre: 26165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 743.669683] Lustre: 26165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 743.678387] Lustre: 26165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 743.687444] Lustre: 26165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.547175] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.568918] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.572040] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 797.768452] Lustre: server umount lustre-OST07c6 complete [ 807.313452] Lustre: server umount lustre-MDT0000 complete [ 825.079308] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 827.876071] Key type lgssc unregistered [ 828.215428] LNet: 28712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.220561] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.246609] LNet: Removed LNI 192.168.204.133@tcp [ 829.035176] Key type .llcrypt unregistered [ 829.037288] Key type ._llcrypt unregistered [ 847.376497] Key type ._llcrypt registered [ 847.379317] Key type .llcrypt registered [ 847.487143] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 848.429730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.435532] alg: No test for adler32 (adler32-zlib) [ 849.490487] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 849.701055] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 851.384187] Key type lgssc registered [ 852.506614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.128643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.146821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.582891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.477776] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 871.908911] LustreError: 29980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.933134] LustreError: 29980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 875.181684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.564460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 880.062309] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 887.696736] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 893.307232] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 900.368913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 900.855468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.005155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 907.760529] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 916.272433] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 937.957610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 937.963681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.972287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 937.972640] LustreError: Skipped 1 previous similar message [ 938.013694] Lustre: Skipped 1 previous similar message [ 939.711773] Lustre: server umount lustre-OST0000 complete [ 947.175375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.186901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.196570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.199405] Lustre: Skipped 1 previous similar message [ 948.196928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 948.204130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.206666] Lustre: Skipped 1 previous similar message [ 950.771837] Lustre: server umount lustre-MDT0000 complete [ 953.319706] LustreError: 29980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 953.358717] LustreError: 29980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 954.332891] LustreError: 29960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875115 with bad export cookie 11805032559422426878 [ 954.356515] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.750387] Lustre: server umount lustre-MDT0001 complete [ 967.660906] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 970.825506] Key type lgssc unregistered [ 971.247592] LNet: 33715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.258235] LNetError: 33715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.280617] LNet: Removed LNI 192.168.204.133@tcp [ 972.479300] Key type .llcrypt unregistered [ 972.484805] Key type ._llcrypt unregistered [ 994.487632] Key type ._llcrypt registered [ 994.489135] Key type .llcrypt registered [ 994.650742] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 995.833070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.853770] alg: No test for adler32 (adler32-zlib) [ 996.922869] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 997.207529] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 998.880220] Key type lgssc registered [ 1000.110723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1007.905905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1007.928445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.501928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.074913] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1019.875350] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.907125] LustreError: 34985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1023.000474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.429918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1028.161547] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1035.821477] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1042.245566] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1049.813743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1050.207130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.336760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1056.992693] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1065.919086] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1079.275450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.277243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.280363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.296964] Lustre: Skipped 1 previous similar message [ 1083.083640] Lustre: server umount lustre-OST0000 complete [ 1089.507507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.519167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.523476] Lustre: Skipped 1 previous similar message [ 1089.537401] Lustre: Skipped 2 previous similar messages [ 1093.372323] Lustre: server umount lustre-MDT0000 complete [ 1094.643481] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.658052] LustreError: 34979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1097.103525] LustreError: 34965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875257 with bad export cookie 2742500314846947160 [ 1097.116294] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1097.118690] LustreError: 34965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1097.500533] Lustre: server umount lustre-MDT0001 complete [ 1109.910482] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1113.073257] Key type lgssc unregistered [ 1113.526300] LNet: 38521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.536539] LNetError: 38521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.604807] LNet: Removed LNI 192.168.204.133@tcp [ 1115.789425] Key type .llcrypt unregistered [ 1115.794339] Key type ._llcrypt unregistered [ 1143.413214] Key type ._llcrypt registered [ 1143.415687] Key type .llcrypt registered [ 1143.555306] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1144.770094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1144.878953] alg: No test for adler32 (adler32-zlib) [ 1145.985165] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1146.276330] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1147.984192] Key type lgssc registered [ 1149.303274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1158.217859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1158.257765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.785348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1164.714951] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1169.892854] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.915825] LustreError: 39939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1175.010487] LustreError: 39938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.163699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1175.505201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1180.911611] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1190.969350] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1197.026993] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1204.677383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.966243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.028804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1212.613102] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1223.885060] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1230.818290] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.825122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.835474] Lustre: Skipped 2 previous similar messages [ 1230.847103] Lustre: Skipped 2 previous similar messages [ 1235.941132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.954864] Lustre: Skipped 3 previous similar messages [ 1236.509502] Lustre: server umount lustre-MDT0000 complete [ 1240.742480] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875401 with bad export cookie 15157627743368986278 [ 1240.747549] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.754740] LustreError: 39918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1241.061733] LustreError: 39934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.066540] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.100570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1241.104377] Lustre: Skipped 1 previous similar message [ 1245.107336] LustreError: 39933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1245.109511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1245.136130] LustreError: 39933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1245.144937] Lustre: Skipped 1 previous similar message [ 1250.219418] LustreError: 41036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.220347] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1250.233564] LustreError: 41036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1250.262253] Lustre: Skipped 1 previous similar message [ 1255.394843] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1255.724877] Lustre: server umount lustre-MDT0001 complete [ 1271.241433] Lustre: server umount lustre-OST0000 complete [ 1290.284718] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1293.627196] Key type lgssc unregistered [ 1293.974720] LNet: 43798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.980389] LNetError: 43798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.998255] LNet: Removed LNI 192.168.204.133@tcp [ 1295.128374] Key type .llcrypt unregistered [ 1295.136158] Key type ._llcrypt unregistered [ 1318.744341] Key type ._llcrypt registered [ 1318.746916] Key type .llcrypt registered [ 1318.867242] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1320.256239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.331079] alg: No test for adler32 (adler32-zlib) [ 1321.495626] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1321.802654] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1323.552264] Key type lgssc registered [ 1324.824947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1331.933062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1331.945717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.322894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.999586] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1343.469480] LustreError: 45214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.482169] LustreError: 45214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1346.159819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.567290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1352.109566] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1361.671960] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1368.396146] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1376.216709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.680776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.748283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1383.605958] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1391.342773] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1397.218450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.224332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.226076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.235427] Lustre: Skipped 1 previous similar message [ 1397.246854] Lustre: Skipped 1 previous similar message [ 1397.892771] Lustre: lustre-OST0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 1402.342483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.353295] Lustre: Skipped 1 previous similar message [ 1407.468526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.478079] Lustre: Skipped 2 previous similar messages [ 1411.040140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1411.232923] Lustre: server umount lustre-OST0000 complete [ 1429.226263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.248328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.249088] Lustre: Skipped 1 previous similar message [ 1429.264830] Lustre: Skipped 2 previous similar messages [ 1435.543446] Lustre: server umount lustre-MDT0000 complete [ 1437.672685] LustreError: 45214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.699698] LustreError: 45214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1439.244765] LustreError: 45195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875600 with bad export cookie 5363013217196007752 [ 1439.262573] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.263894] LustreError: 45195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1445.729517] Lustre: server umount lustre-MDT0001 complete [ 1456.500679] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1459.435938] Key type lgssc unregistered [ 1459.793701] LNet: 48921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.806930] LNetError: 48921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.829899] LNet: Removed LNI 192.168.204.133@tcp [ 1460.809773] Key type .llcrypt unregistered [ 1460.812293] Key type ._llcrypt unregistered [ 1482.848625] Key type ._llcrypt registered [ 1482.854492] Key type .llcrypt registered [ 1482.972380] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1483.829457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1483.852727] alg: No test for adler32 (adler32-zlib) [ 1484.927504] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1485.158599] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1486.817277] Key type lgssc registered [ 1487.776199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.057760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.070172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.488551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.620326] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1506.786149] LustreError: 50316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.797598] LustreError: 50316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1508.930363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.296869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.917600] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1522.390192] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1528.746265] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1535.665985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.133245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.299922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1543.250623] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1551.481575] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1560.442313] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1564.062326] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1576.168232] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 9h55m34s [ 1577.761945] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1580.515365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1580.529659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1580.540611] Lustre: Skipped 2 previous similar messages [ 1580.554954] Lustre: Skipped 2 previous similar messages [ 1581.610703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1581.622180] Lustre: Skipped 2 previous similar messages [ 1584.239740] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 1587.686567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.700674] Lustre: Skipped 3 previous similar messages [ 1593.845824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1593.858704] Lustre: Skipped 3 previous similar messages [ 1594.336215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1594.730793] Lustre: server umount lustre-MDT0000 complete [ 1597.535880] LustreError: 50312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1598.610252] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1599.972217] LustreError: 50317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.002051] LustreError: 50317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1603.678904] LustreError: 50312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.372858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.521530] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1605.792209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.841385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1606.835706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.241153] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1613.622326] Lustre: DEBUG MARKER: Wait for df (36730) ... [ 1615.269664] Lustre: DEBUG MARKER: done [ 1621.473807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1621.474459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1621.487546] LustreError: Skipped 1 previous similar message [ 1621.492251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1621.524651] Lustre: Skipped 1 previous similar message [ 1625.760424] Lustre: server umount lustre-OST0000 complete [ 1632.737768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.740051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1632.760708] Lustre: Skipped 1 previous similar message [ 1636.559582] Lustre: server umount lustre-MDT0000 complete [ 1639.911530] LustreError: 50316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.932794] LustreError: 50316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1640.873697] LustreError: 50296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875801 with bad export cookie 7875771239927966547 [ 1640.874700] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1640.885220] LustreError: 50296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1641.250161] Lustre: server umount lustre-MDT0001 complete [ 1655.288924] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1658.663435] Key type lgssc unregistered [ 1658.989933] LNet: 55242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.994794] LNetError: 55242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.006783] LNet: Removed LNI 192.168.204.133@tcp [ 1660.931538] Key type .llcrypt unregistered [ 1660.934772] Key type ._llcrypt unregistered [ 1704.108527] Key type ._llcrypt registered [ 1704.114336] Key type .llcrypt registered [ 1704.256686] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1705.346500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1705.502370] alg: No test for adler32 (adler32-zlib) [ 1706.535788] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1706.776417] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1708.480280] Key type lgssc registered [ 1709.899546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1717.784965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1717.819243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.114286] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1719.148682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1719.400568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.379078] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1733.387479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.526719] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1733.565396] Lustre: 57447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1733.879634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1737.936427] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1745.367811] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1751.354189] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1758.449576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1758.589576] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1758.759482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.664345] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1768.602526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1771.649788] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1778.432731] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 9h58m57s [ 1779.728943] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1786.168247] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1788.256936] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1788.308174] LustreError: 57455:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1808.532794] Lustre: lustre-MDT0000: Client 6f602e90-63ea-45a3-81a4-a4ae2d93d673 (at 192.168.204.33@tcp) reconnecting [ 1828.312920] LustreError: 57455:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1830.814409] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1833.425572] Lustre: DEBUG MARKER: done [ 1840.101340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.121145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.133417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.613960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1840.628479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.639163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.739461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.748889] Lustre: Skipped 1 previous similar message [ 1846.001665] Lustre: server umount lustre-OST0000 complete [ 1850.864190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.888445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.899278] Lustre: Skipped 1 previous similar message [ 1857.832812] Lustre: server umount lustre-MDT0000 complete [ 1861.390218] LustreError: 59307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876022 with bad export cookie 14839891810422252255 [ 1861.393242] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1861.397136] LustreError: 59307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1861.745562] Lustre: server umount lustre-MDT0001 complete [ 1873.466465] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 1876.634963] Key type lgssc unregistered [ 1876.978933] LNet: 61775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.987538] LNetError: 61775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.020414] LNet: Removed LNI 192.168.204.133@tcp [ 1878.148297] Key type .llcrypt unregistered [ 1878.150707] Key type ._llcrypt unregistered [ 1922.419836] Key type ._llcrypt registered [ 1922.423720] Key type .llcrypt registered [ 1922.575977] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 1924.156927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.181421] alg: No test for adler32 (adler32-zlib) [ 1925.235346] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1925.449756] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 1927.112543] Key type lgssc registered [ 1928.138413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1931.356136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1931.374527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.613082] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1932.639932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1932.805879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1933.817900] Lustre: Failing over lustre-MDT0000 [ 1934.195369] Lustre: server umount lustre-MDT0000 complete [ 1940.595847] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:01:38 (1788876098) [ 1947.810320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.231948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.118175] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1961.592037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.688831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1961.720865] Lustre: 64586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1962.008248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1966.285962] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1974.758573] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1980.696720] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1987.746210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1988.006557] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1988.236583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.487877] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 1997.434052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2002.714502] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2007.630481] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2018.273966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.278813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.296439] Lustre: Skipped 2 previous similar messages [ 2018.314154] Lustre: Skipped 2 previous similar messages [ 2020.277684] Lustre: server umount lustre-MDT0000 complete [ 2023.248083] LustreError: 64576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876184 with bad export cookie 14199896944246341806 [ 2023.256304] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.260322] LustreError: 64576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2023.398433] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.542287] Lustre: server umount lustre-MDT0001 complete [ 2026.617418] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2031.427139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2040.457888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.998511] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc5102b0958f54991 [ 2049.022498] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo) [ 2049.403750] LustreError: 66437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2049.421834] LustreError: 66437:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2049.493943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.610598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2053.482058] LustreError: 66438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.335655] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2058.721662] LustreError: 66437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.853191] LustreError: 66438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.879290] LustreError: 66438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2064.000350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.399852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.237334] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2069.490773] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2073.637476] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.927992] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.236597] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.050261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.058089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.066034] Lustre: Skipped 2 previous similar messages [ 2094.084076] Lustre: Skipped 2 previous similar messages [ 2097.946697] Lustre: server umount lustre-MDT0000 complete [ 2099.175730] LustreError: 68360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2099.210219] LustreError: 68360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2102.346610] LustreError: 65333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876263 with bad export cookie 14199896944246344081 [ 2102.354756] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.363401] LustreError: 65333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2102.871042] Lustre: server umount lustre-MDT0001 complete [ 2110.770100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2120.672689] Lustre: 63508:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788876265/real 1788876265] req@000000005d26ecc4 x1875772530822016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788876281 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.720146] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.794912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.489861] Lustre: 63509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788876268/real 1788876268] req@000000003416d3ad x1875772530822144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788876284 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.793885] Lustre: 63507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788876270/real 1788876270] req@000000008aa5bb8e x1875772530822272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788876286 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2128.268688] LustreError: 66438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.360144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.418326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2129.382876] Lustre: 63507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788876273/real 1788876273] req@000000000f580502 x1875772530822528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788876289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2132.903654] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2133.475339] Lustre: 63508:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788876278/real 1788876278] req@0000000066febca2 x1875772530822656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788876294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2141.733085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.555933] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2147.323135] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2156.247472] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2162.233814] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2169.839304] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.847143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2169.864915] Lustre: Skipped 2 previous similar messages [ 2169.871791] Lustre: Skipped 1 previous similar message [ 2174.950982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.968332] Lustre: Skipped 2 previous similar messages [ 2175.483644] Lustre: server umount lustre-MDT0000 complete [ 2179.575690] LustreError: 65333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876340 with bad export cookie 14199896944246345460 [ 2179.581131] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.585816] LustreError: 65333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2179.840949] Lustre: server umount lustre-MDT0001 complete [ 2186.239885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2189.897386] Lustre: DEBUG MARKER: files compared the same [ 2202.230760] Lustre: server umount lustre-OST0000 complete [ 2217.212921] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2219.805909] Key type lgssc unregistered [ 2220.069870] LNet: 74315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.080555] LNetError: 74315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.106816] LNet: Removed LNI 192.168.204.133@tcp [ 2220.910443] Key type .llcrypt unregistered [ 2220.914278] Key type ._llcrypt unregistered [ 2241.965409] Key type ._llcrypt registered [ 2241.967995] Key type .llcrypt registered [ 2242.135403] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2243.129703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2243.163024] alg: No test for adler32 (adler32-zlib) [ 2244.171481] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2244.379621] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2246.056229] Key type lgssc registered [ 2247.056935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2254.677593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2254.705986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.342227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.353341] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2266.597213] LustreError: 75709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.610047] LustreError: 75709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2268.055724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2268.382750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2272.898274] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2281.147269] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2286.069827] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2291.915928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2292.182042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.196377] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug +malloc all [ 2301.422227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2305.135810] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2313.703886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2313.715300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.744633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.769138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.775617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.782751] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2317.952810] Lustre: server umount lustre-OST0000 complete [ 2321.889320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.905724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.912932] Lustre: Skipped 2 previous similar messages [ 2327.008943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.022304] Lustre: Skipped 1 previous similar message [ 2327.440618] Lustre: server umount lustre-MDT0000 complete [ 2331.000907] LustreError: 75692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876491 with bad export cookie 9620428862122790093 [ 2331.004574] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2331.011329] LustreError: 75692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2331.265073] Lustre: server umount lustre-MDT0001 complete [ 2341.295975] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2343.586564] Key type lgssc unregistered [ 2343.880977] LNet: 79243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.887871] LNetError: 79243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.902781] LNet: Removed LNI 192.168.204.133@tcp [ 2344.714861] Key type .llcrypt unregistered [ 2344.716469] Key type ._llcrypt unregistered [ 2365.568991] Key type ._llcrypt registered [ 2365.573806] Key type .llcrypt registered [ 2365.703819] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2366.473792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.582640] alg: No test for adler32 (adler32-zlib) [ 2367.635968] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2367.861067] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2369.514523] Key type lgssc registered [ 2370.598140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2377.639610] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2377.663576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2393.185816] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2398.305318] LustreError: 80609:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [ 2424.928483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.190227] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2438.695912] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2446.368756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.463543] Lustre: *** cfs_fail_loc=706, val=0*** [ 2446.747187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.836986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2451.181359] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2451.940270] LustreError: 80634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.951393] LustreError: 80634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2457.058936] LustreError: 80635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.348876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.702337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.592681] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2470.648603] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.334493] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2480.099533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2480.112950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.142295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.125211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2481.135506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.407468] Lustre: server umount lustre-OST0000 complete [ 2490.337704] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2490.349577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.366706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.374208] Lustre: Skipped 1 previous similar message [ 2496.482724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2496.490108] Lustre: Skipped 3 previous similar messages [ 2501.601223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2501.609506] Lustre: Skipped 1 previous similar message [ 2502.112168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2502.224040] Lustre: server umount lustre-MDT0000 complete [ 2505.386488] LustreError: 80616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876666 with bad export cookie 2780791504170611795 [ 2505.397042] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.398268] LustreError: 80616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2505.712367] Lustre: server umount lustre-MDT0001 complete [ 2515.217306] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2517.896624] Key type lgssc unregistered [ 2518.151536] LNet: 84221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.162483] LNetError: 84221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.185284] LNet: Removed LNI 192.168.204.133@tcp [ 2518.950094] Key type .llcrypt unregistered [ 2518.953238] Key type ._llcrypt unregistered [ 2543.131498] Key type ._llcrypt registered [ 2543.136600] Key type .llcrypt registered [ 2543.259811] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2544.417900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.448813] alg: No test for adler32 (adler32-zlib) [ 2545.450646] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2545.656414] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2547.329673] Key type lgssc registered [ 2548.395926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2555.967707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2555.981036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2561.217605] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2568.436803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.831466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.562807] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2574.306253] LustreError: 86322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.319188] LustreError: 86322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2578.410089] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.435730] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2579.982585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.266838] LustreError: 86322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.284039] LustreError: 86322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2580.296863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2585.571262] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.580823] LustreError: 86326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2585.640183] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2590.695603] LustreError: 86321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.714548] LustreError: 86321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2593.520975] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.538582] Lustre: MGS: Client c2110963-a5e1-4971-95a6-73365c3de555 (at 0@lo) reconnecting [ 2593.560701] LustreError: 86336:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2593.795673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.861496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2597.575552] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2614.246256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2614.256552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2614.272706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.265971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.278118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.281553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.303536] Lustre: Skipped 1 previous similar message [ 2617.534576] Lustre: server umount lustre-OST0000 complete [ 2625.507963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2625.516855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2625.529876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.626724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2630.632762] Lustre: Skipped 2 previous similar messages [ 2635.232125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2635.334647] Lustre: server umount lustre-MDT0000 complete [ 2635.746174] LustreError: 86325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.758453] LustreError: 86325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2638.486508] Lustre: server umount lustre-MDT0001 complete [ 2641.664495] Lustre: server umount MGS complete [ 2650.922954] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:13:29 (1788876809) [ 2673.890722] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 2682.437045] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2713.608170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.096371] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2728.069107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.461582] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2728.495470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2728.578330] Lustre: lustre-MDT0001: new disk, initializing [ 2728.659247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2728.680449] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2728.689992] Lustre: 93514:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2728.698388] LustreError: 93514:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2733.061856] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2742.973382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2743.210297] Lustre: lustre-OST0000: new disk, initializing [ 2743.214261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2743.219822] Lustre: 94280:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2743.293699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2748.729454] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2754.593345] Lustre: 94840:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000b22920af x1875773180911616/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788876931 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2757.480885] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2764.690446] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.712961] Lustre: MGS: Client 6fbdddc1-2957-405d-baf5-b12da9dcb931 (at 0@lo) reconnecting [ 2764.728272] LustreError: 93500:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2764.760375] Lustre: 92782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2764.794168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2764.882594] Lustre: lustre-MDT0000: new disk, initializing [ 2765.029526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.062743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2765.197819] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2766.117868] Lustre: 95350:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d47dbbf8 x1875773180918656/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788876942 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2769.608563] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2775.028278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2775.036912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2775.349525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2800.609573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2800.618147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2800.630027] Lustre: Skipped 1 previous similar message [ 2800.637131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2805.823552] Lustre: server umount lustre-OST0000 complete [ 2814.434888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2814.446873] LustreError: Skipped 1 previous similar message [ 2814.451988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.459911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2814.480881] Lustre: Skipped 2 previous similar messages [ 2814.496171] Lustre: Skipped 4 previous similar messages [ 2816.202442] Lustre: server umount lustre-MDT0000 complete [ 2819.556696] LustreError: 93489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.579569] LustreError: 93489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2834.400319] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2834.634757] Lustre: server umount lustre-MDT0001 complete [ 2838.204266] Lustre: server umount MGS complete [ 2848.991995] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:16:47 (1788877007) [ 2867.908665] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2870.284351] Key type lgssc unregistered [ 2870.585631] LNet: 97849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2870.594343] LNetError: 97849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2870.614728] LNet: Removed LNI 192.168.204.133@tcp [ 2871.342775] Key type .llcrypt unregistered [ 2871.346547] Key type ._llcrypt unregistered [ 2888.048838] Key type ._llcrypt registered [ 2888.051938] Key type .llcrypt registered [ 2888.127887] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 2888.906391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2889.047258] alg: No test for adler32 (adler32-zlib) [ 2890.180371] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2890.484711] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 2892.153375] Key type lgssc registered [ 2893.253711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2900.598447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2900.625159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.138459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.973330] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2911.716835] LustreError: 99070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2911.735325] LustreError: 99070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2915.453629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.893873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.959847] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2927.931465] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2933.775741] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2940.462993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.803718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.851803] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 2946.919072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2953.776314] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2965.474967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.482603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2965.482726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2965.491932] Lustre: Skipped 1 previous similar message [ 2965.508748] Lustre: Skipped 1 previous similar message [ 2967.066918] Lustre: server umount lustre-OST0000 complete [ 2970.596082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.597078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.605334] Lustre: Skipped 1 previous similar message [ 2970.614875] Lustre: Skipped 1 previous similar message [ 2975.715389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2975.721120] Lustre: Skipped 1 previous similar message [ 2976.818363] Lustre: server umount lustre-MDT0000 complete [ 2980.106640] LustreError: 99049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877140 with bad export cookie 10429481671854084687 [ 2980.109610] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.119239] LustreError: 99049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2980.401928] Lustre: server umount lustre-MDT0001 complete [ 2990.139705] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 2992.479051] Key type lgssc unregistered [ 2992.739831] LNet: 102601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2992.750504] LNetError: 102601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2992.766221] LNet: Removed LNI 192.168.204.133@tcp [ 2993.456469] Key type .llcrypt unregistered [ 2993.460347] Key type ._llcrypt unregistered [ 3007.862921] Key type ._llcrypt registered [ 3007.866842] Key type .llcrypt registered [ 3008.010393] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3008.893419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3008.915991] alg: No test for adler32 (adler32-zlib) [ 3009.973876] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3010.216118] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3011.912350] Key type lgssc registered [ 3012.929513] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3018.067282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3018.073543] LustreError: 103673:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3018.111225] LustreError: 103680:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3018.121030] LustreError: 103673:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4966 [ 3018.126907] LustreError: 103673:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3018.133802] LustreError: 103673:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3018.142749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.492173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3025.663572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.884114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3028.844553] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3033.005471] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing lsmod [ 3035.222876] LustreError: 104883:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3035.228842] LustreError: 104886:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3035.234122] LustreError: 104883:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3035.239238] LustreError: 104883:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3035.248880] LustreError: 104883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3035.256689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3035.610882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3041.249257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.256362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.266064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.762241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.770214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.775474] Lustre: Skipped 1 previous similar message [ 3044.454069] Lustre: server umount lustre-MDT0000 complete [ 3046.882063] LustreError: 103717:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.892197] LustreError: 103717:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3047.701780] LustreError: 103691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877208 with bad export cookie 9189214459443601003 [ 3047.714058] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.714831] LustreError: 103691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3047.929472] Lustre: server umount lustre-MDT0001 complete [ 3060.369153] Lustre: server umount lustre-OST0000 complete [ 3067.370401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3067.677238] LustreError: 105949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.747517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.416814] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3072.999136] LustreError: 105950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.115609] LustreError: 105949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.293752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.567994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.177975] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3088.139717] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3093.780409] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3099.819326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3100.153309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.966438] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3111.598567] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3112.441208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3122.663444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.665507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.669726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.684565] Lustre: Skipped 1 previous similar message [ 3122.693310] Lustre: Skipped 2 previous similar messages [ 3127.199240] Lustre: server umount lustre-OST0000 complete [ 3132.898540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.903151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.914095] Lustre: Skipped 1 previous similar message [ 3132.920859] Lustre: Skipped 1 previous similar message [ 3136.791768] Lustre: server umount lustre-MDT0000 complete [ 3138.019373] LustreError: 107044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.032377] LustreError: 107044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3140.198709] LustreError: 105931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877301 with bad export cookie 9189214459443601703 [ 3140.199242] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3140.208528] LustreError: 105931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3140.541873] Lustre: server umount lustre-MDT0001 complete [ 3149.419744] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3151.973740] Key type lgssc unregistered [ 3152.259521] LNet: 109484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.267277] LNetError: 109484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.280185] LNet: Removed LNI 192.168.204.133@tcp [ 3153.030815] Key type .llcrypt unregistered [ 3153.033640] Key type ._llcrypt unregistered [ 3174.379764] Key type ._llcrypt registered [ 3174.385449] Key type .llcrypt registered [ 3174.471365] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3175.204380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3175.291020] alg: No test for adler32 (adler32-zlib) [ 3176.328286] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3176.508274] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3178.152360] Key type lgssc registered [ 3179.101404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3185.317526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3185.337305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.747464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.632893] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3196.901839] LustreError: 110898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.925182] LustreError: 110898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3197.820765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.149488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.849509] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3208.613719] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3213.431755] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3219.177739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.405481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.758327] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3231.187169] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3231.798495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3236.841645] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3239.890642] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3245.576204] Lustre: server umount lustre-OST0000 complete [ 3249.633491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3249.645668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.654624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.196080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.196739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3257.315636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3257.326360] Lustre: Skipped 2 previous similar messages [ 3258.381645] Lustre: server umount lustre-MDT0000 complete [ 3261.262145] LustreError: 112742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877422 with bad export cookie 3256047556802112195 [ 3261.265173] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.271100] LustreError: 112742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3271.765191] Lustre: server umount lustre-MDT0001 complete [ 3280.038460] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3282.097443] Key type lgssc unregistered [ 3282.284166] LNet: 114535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3282.288489] LNetError: 114535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3282.301208] LNet: Removed LNI 192.168.204.133@tcp [ 3282.899980] Key type .llcrypt unregistered [ 3282.903679] Key type ._llcrypt unregistered [ 3296.571353] Key type ._llcrypt registered [ 3296.573293] Key type .llcrypt registered [ 3296.661419] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3297.457702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3297.473805] alg: No test for adler32 (adler32-zlib) [ 3298.460691] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3298.718753] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3300.376171] Key type lgssc registered [ 3301.195206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3306.723430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3306.747744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.145458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.730747] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3317.732443] LustreError: 115741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.740165] LustreError: 115741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3318.132829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3318.446841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.225419] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3326.697517] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3330.526496] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3335.936432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3336.204368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.308538] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3347.219910] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3352.624921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3363.818532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3363.836869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3363.869592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3366.023878] Lustre: server umount lustre-OST0000 complete [ 3373.032285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.055049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.061096] Lustre: Skipped 1 previous similar message [ 3378.146262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3378.161498] Lustre: Skipped 1 previous similar message [ 3382.249702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.270420] Lustre: Skipped 1 previous similar message [ 3386.336299] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3387.366254] LustreError: 115741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.396865] LustreError: 115741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3387.772740] Lustre: server umount lustre-MDT0000 complete [ 3392.483576] LustreError: 115738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.534486] LustreError: 115738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3397.292563] LustreError: 115722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877558 with bad export cookie 4368510546520376576 [ 3397.303858] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3397.328406] LustreError: 115722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3399.425994] Lustre: server umount lustre-MDT0001 complete [ 3415.912590] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3420.232782] Key type lgssc unregistered [ 3420.701714] LNet: 119313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3420.706130] LNetError: 119313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3421.743871] LNet: Removed LNI 192.168.204.133@tcp [ 3423.484685] Key type .llcrypt unregistered [ 3423.487753] Key type ._llcrypt unregistered [ 3456.247896] Key type ._llcrypt registered [ 3456.249925] Key type .llcrypt registered [ 3456.344533] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3457.455705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3457.510141] alg: No test for adler32 (adler32-zlib) [ 3459.038357] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3459.530571] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3461.240259] Key type lgssc registered [ 3462.587979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.160572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.179072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.521979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.361244] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3484.134716] LustreError: 120729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.149868] LustreError: 120729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3488.299876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.657928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3493.904408] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3503.314607] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3510.147320] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3516.788722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3517.092113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.151943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3523.992633] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3532.853352] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3539.703390] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3543.548911] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3543.557699] Lustre: Skipped 2 previous similar messages [ 3548.189493] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3548.194584] Lustre: Skipped 2 previous similar messages [ 3548.771758] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3548.805234] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3559.938861] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3559.947489] Lustre: Skipped 2 previous similar messages [ 3568.714995] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.33@tcp 192.168.0.[2,10]@tcp [ 3572.228142] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.33/24@tcp in log lustre-MDT0000 [ 3572.235666] Lustre: Skipped 2 previous similar messages [ 3578.454146] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.33/24@tcp [ 3578.471479] Lustre: Skipped 1 previous similar message [ 3588.091650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3588.094353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.100439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.106057] LustreError: Skipped 3 previous similar messages [ 3588.146450] Lustre: Skipped 1 previous similar message [ 3593.188907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.203400] Lustre: Skipped 2 previous similar messages [ 3593.576699] Lustre: server umount lustre-OST0000 complete [ 3598.307671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.321706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.330254] Lustre: Skipped 1 previous similar message [ 3598.343181] Lustre: Skipped 1 previous similar message [ 3603.416205] Lustre: server umount lustre-MDT0000 complete [ 3603.426645] LustreError: 122020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.448111] LustreError: 122020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3607.439672] LustreError: 120711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877768 with bad export cookie 18418100225898336051 [ 3607.445793] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3607.456627] LustreError: 120711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3608.045853] Lustre: server umount lustre-MDT0001 complete [ 3620.490515] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3624.079725] Key type lgssc unregistered [ 3624.421803] LNet: 125618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.447386] LNetError: 125618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.463754] LNet: Removed LNI 192.168.204.133@tcp [ 3625.622942] Key type .llcrypt unregistered [ 3625.625639] Key type ._llcrypt unregistered [ 3662.865398] Key type ._llcrypt registered [ 3662.871616] Key type .llcrypt registered [ 3663.050359] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3664.193504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3664.230117] alg: No test for adler32 (adler32-zlib) [ 3665.468471] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3665.780574] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3667.448148] Key type lgssc registered [ 3668.307679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3675.917201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3675.944744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3677.407808] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3677.447835] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.33/24@tcp [ 3677.448369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3682.934705] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3687.912263] LustreError: 127179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.942153] LustreError: 127179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3692.307868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3692.683662] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3692.724160] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.33/24@tcp [ 3692.823151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3698.244480] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3706.843951] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3713.931694] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3720.668412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3720.915929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.017719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3726.957704] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3734.443040] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.833765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.842257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.863624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.936713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.946073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.966761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.264826] Lustre: server umount lustre-OST0000 complete [ 3759.588989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.600630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.610114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.617064] Lustre: Skipped 1 previous similar message [ 3763.173570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.183683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.454773] Lustre: server umount lustre-MDT0000 complete [ 3768.289660] LustreError: 129679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3768.301412] LustreError: 129679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3768.671089] LustreError: 127160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877929 with bad export cookie 16552048878923975032 [ 3768.679366] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3768.686323] LustreError: 127160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3769.121637] Lustre: server umount lustre-MDT0001 complete [ 3782.426214] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 3785.736794] Key type lgssc unregistered [ 3786.119803] LNet: 130769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3786.132938] LNetError: 130769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3786.149937] LNet: Removed LNI 192.168.204.133@tcp [ 3787.345528] Key type .llcrypt unregistered [ 3787.349541] Key type ._llcrypt unregistered [ 3815.739836] Key type ._llcrypt registered [ 3815.743289] Key type .llcrypt registered [ 3815.888210] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 3831.298399] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 3832.564794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3832.602142] alg: No test for adler32 (adler32-zlib) [ 3833.681492] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 3833.935441] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 3835.616185] Key type lgssc registered [ 3836.825668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3870.709124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3870.729580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3871.935761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3871.966803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3872.071921] Lustre: lustre-MDT0000: new disk, initializing [ 3872.151100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.180331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3876.596398] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3887.385665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.516583] Lustre: 134167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3887.565894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3887.569578] Lustre: Skipped 1 previous similar message [ 3887.630863] Lustre: lustre-MDT0001: new disk, initializing [ 3887.707832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3887.739770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3887.746248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3892.630898] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3902.762418] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3908.537401] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3915.846163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.116488] Lustre: lustre-OST0000: new disk, initializing [ 3916.120549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3916.133552] Lustre: 136134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3916.210324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3922.361785] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3924.052281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3924.060395] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3924.190202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3935.819075] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3943.719075] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3943.997105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3948.908884] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3949.236390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3955.169153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.175069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.179637] LustreError: Skipped 1 previous similar message [ 3955.180970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.207389] Lustre: Skipped 1 previous similar message [ 3957.384446] Lustre: server umount lustre-OST0000 complete [ 3964.385836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3964.390949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.398193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3964.401202] Lustre: Skipped 1 previous similar message [ 3965.412636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3965.412805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3965.423688] Lustre: Skipped 1 previous similar message [ 3967.003886] Lustre: server umount lustre-MDT0000 complete [ 3970.622522] LustreError: 134962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878131 with bad export cookie 4086038087914843426 [ 3970.623650] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.630041] LustreError: 134962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3970.845797] Lustre: server umount lustre-MDT0001 complete [ 3979.934624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.318726] LustreError: 138350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.342200] LustreError: 138350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3980.379903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3985.161115] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 3985.389507] LustreError: 138351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3990.502400] LustreError: 138351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.894202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.112731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.634780] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4006.061391] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4012.101928] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.663238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.946827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4024.214618] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4030.721931] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4034.592909] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4034.819777] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4038.307755] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4039.554649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 4047.016963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.174134] Lustre: lustre-OST0001: new disk, initializing [ 4047.177372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4047.182071] Lustre: 141731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4047.256640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.842596] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4054.626871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4054.637981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4054.728932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4060.784917] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4060.993550] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4064.398233] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4064.562621] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4069.359226] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98de05a7c000.ost_server_uuid 50 [ 4070.850266] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98de05a7c000.ost_server_uuid in FULL state after 0 sec [ 4080.610627] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4080.631722] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.670952] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4080.682750] Lustre: Skipped 1 previous similar message [ 4081.374328] Lustre: server umount lustre-OST0001 complete [ 4084.705493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4084.713820] LustreError: Skipped 1 previous similar message [ 4084.719788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.731796] Lustre: Skipped 1 previous similar message [ 4084.735128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4085.732297] LustreError: 140171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4085.744087] LustreError: 140171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4090.875285] Lustre: server umount lustre-OST0000 complete [ 4095.974941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.981510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4095.987302] Lustre: Skipped 2 previous similar messages [ 4096.000655] Lustre: Skipped 3 previous similar messages [ 4108.257172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4108.404802] Lustre: server umount lustre-MDT0000 complete [ 4110.307831] LustreError: 139078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.329892] LustreError: 139078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4111.637990] LustreError: 138333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878272 with bad export cookie 4086038087914844308 [ 4111.641679] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4111.645517] LustreError: 138333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4111.914070] Lustre: server umount lustre-MDT0001 complete [ 4127.373207] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4129.279311] Key type lgssc unregistered [ 4129.520512] LNet: 144733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4129.529706] LNetError: 144733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4129.542435] LNet: Removed LNI 192.168.204.133@tcp [ 4130.204455] Key type .llcrypt unregistered [ 4130.206220] Key type ._llcrypt unregistered [ 4158.784754] Key type ._llcrypt registered [ 4158.788148] Key type .llcrypt registered [ 4158.870876] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4171.019949] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4172.001283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4172.025918] alg: No test for adler32 (adler32-zlib) [ 4173.051551] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 4173.250438] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4174.913493] Key type lgssc registered [ 4175.763119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4202.731886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4202.766163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.997616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4204.032432] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4204.078548] Lustre: lustre-MDT0000: new disk, initializing [ 4204.148047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4204.163350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4207.740250] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4216.688418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.805237] Lustre: 148861:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4216.860591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4216.869472] Lustre: Skipped 1 previous similar message [ 4216.947294] Lustre: lustre-MDT0001: new disk, initializing [ 4217.026486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4217.057064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4217.065290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4220.885473] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4228.925440] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4233.395339] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4238.615656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.819948] Lustre: lustre-OST0000: new disk, initializing [ 4238.826831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4238.836157] Lustre: 150829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4238.915866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.498544] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4248.083587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4248.094197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4248.186928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4250.225698] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4254.312241] Lustre: Failing over lustre-OST0000 [ 4254.393400] Lustre: server umount lustre-OST0000 complete [ 4270.078580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.298379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.308332] Lustre: Skipped 1 previous similar message [ 4270.321717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.332432] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4272.608575] Lustre: 147085:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788878417/real 1788878417] req@0000000004885597 x1875774887982976/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788878433 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4273.277462] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4274.105759] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4274.400365] Lustre: 147084:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788878419/real 1788878419] req@00000000156e7160 x1875774887984384/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788878435 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4275.699123] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4275.699982] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4276.386586] Lustre: Failing over lustre-MDT0000 [ 4276.561797] Lustre: server umount lustre-MDT0000 complete [ 4278.764608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4278.770263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.456163] Lustre: 147086:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788878424/real 1788878424] req@000000001c357a49 x1875774887985280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788878440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4279.470215] Lustre: 147086:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4280.806063] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.485691] LustreError: 148870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.499134] LustreError: 148870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4284.896787] Lustre: 147086:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788878429/real 1788878429] req@0000000026bb8ddc x1875774887986304/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788878445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4285.922986] LustreError: 148874:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.935340] LustreError: 148874:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4288.604043] LustreError: 148868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.049639] LustreError: 148873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4291.060833] LustreError: 148873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4292.850906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.932645] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4292.940803] LustreError: 153048:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.204.133@tcp: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000e51928f4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4292.952984] Lustre: 148865:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.133@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4293.106472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.146976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4293.728773] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4295.899221] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4298.222958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4298.235350] Lustre: Skipped 1 previous similar message [ 4298.254290] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4298.298728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4303.335452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.340796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.342691] Lustre: Skipped 2 previous similar messages [ 4303.355255] Lustre: Skipped 1 previous similar message [ 4307.500474] Lustre: server umount lustre-OST0000 complete [ 4313.570557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.571219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4313.575540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4313.580307] Lustre: Skipped 1 previous similar message [ 4318.690336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.693018] Lustre: Skipped 1 previous similar message [ 4323.811615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.320630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4324.416864] Lustre: server umount lustre-MDT0000 complete [ 4326.893245] LustreError: 148853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878487 with bad export cookie 7990466527182352332 [ 4326.896068] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.898851] LustreError: 148853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4327.105657] Lustre: server umount lustre-MDT0001 complete [ 4333.627439] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4335.343771] Key type lgssc unregistered [ 4335.526656] LNet: 154479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4335.533476] LNetError: 154479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4335.547228] LNet: Removed LNI 192.168.204.133@tcp [ 4336.082667] Key type .llcrypt unregistered [ 4336.084376] Key type ._llcrypt unregistered [ 4364.293682] Key type ._llcrypt registered [ 4364.295625] Key type .llcrypt registered [ 4364.386531] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4365.064696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4365.133077] alg: No test for adler32 (adler32-zlib) [ 4366.063688] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 4366.198100] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4367.818910] Key type lgssc registered [ 4368.506741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4373.389559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4373.403413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.667732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4377.446812] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4383.768343] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4384.226314] LustreError: 156002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4384.239198] LustreError: 156002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4386.358494] LustreError: 156030:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4386.364134] LustreError: 156030:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4386.368883] LustreError: 156030:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 4388.593057] Lustre: server umount lustre-MDT0000 complete [ 4397.815113] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4403.476668] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4424.732090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.908265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4424.933315] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4424.987672] Lustre: lustre-MDT0000: new disk, initializing [ 4425.051279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4425.066426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4427.593488] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4434.423442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.490249] Lustre: 159337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4434.528825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4434.533265] Lustre: Skipped 1 previous similar message [ 4434.589675] Lustre: lustre-MDT0001: new disk, initializing [ 4434.670549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.697317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4434.703063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4437.521431] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4443.945937] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4447.855740] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4452.280931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4452.453772] Lustre: lustre-OST0000: new disk, initializing [ 4452.461686] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4452.468596] Lustre: 161308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4452.547925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.350539] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4460.045806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4460.054356] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4460.111068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4461.816118] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4465.393754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4466.364347] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:43:45 (1788878625) [ 4613.601155] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.601785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.610474] Lustre: Skipped 2 previous similar messages [ 4613.619995] Lustre: Skipped 2 previous similar messages [ 4618.723301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.727084] Lustre: Skipped 2 previous similar messages [ 4619.205510] Lustre: server umount lustre-MDT0000 complete [ 4621.259720] LustreError: 159332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878782 with bad export cookie 15639846187967032231 [ 4621.260902] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.264373] LustreError: 159332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4623.842415] LustreError: 159347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.842527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.859614] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4626.848618] LustreError: 159345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.848624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4627.510863] Lustre: server umount lustre-MDT0001 complete [ 4643.808108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4643.888467] Lustre: server umount lustre-OST0000 complete [ 4649.544578] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4655.753429] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4678.463732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4678.626564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4678.644714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4678.696400] Lustre: lustre-MDT0000: new disk, initializing [ 4678.755896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.770730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4681.466987] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4688.141781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.206616] Lustre: 165955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4688.246329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4688.249829] Lustre: Skipped 1 previous similar message [ 4688.299395] Lustre: lustre-MDT0001: new disk, initializing [ 4688.369206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4688.403210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4688.407312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4690.902480] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4697.569364] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4701.245732] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4705.569962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4705.738883] Lustre: lustre-OST0000: new disk, initializing [ 4705.742925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4705.750348] Lustre: 167924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4705.798983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.145298] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4712.993499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4713.000114] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4713.056426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4714.756697] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4717.486411] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4717.628248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4720.567578] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4720.729859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4723.169402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4723.177485] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4723.181061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4723.183080] Lustre: Skipped 1 previous similar message [ 4723.190367] Lustre: Skipped 1 previous similar message [ 4728.249276] Lustre: server umount lustre-OST0000 complete [ 4733.412623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4733.414746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4733.420198] Lustre: Skipped 1 previous similar message [ 4736.843122] Lustre: server umount lustre-MDT0000 complete [ 4738.529716] LustreError: 165966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.542646] LustreError: 165966:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4739.324313] LustreError: 165946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878900 with bad export cookie 15639846187967191047 [ 4739.330284] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4739.333468] LustreError: 165946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4739.590568] Lustre: server umount lustre-MDT0001 complete [ 4744.878369] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:48:23 (1788878903) [ 4752.626917] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4757.724570] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4779.385908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.543180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4779.558804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4779.602768] Lustre: lustre-MDT0000: new disk, initializing [ 4779.657776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.669402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4782.263543] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4789.569689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.653892] Lustre: 172672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4789.879751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4792.155222] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4797.924932] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4801.124738] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4804.868583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4805.034451] Lustre: 174652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4808.745256] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4811.332371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4811.408318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4815.124938] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4821.988413] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4821.989069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4821.994991] LustreError: Skipped 2 previous similar messages [ 4822.003758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4822.008463] Lustre: Skipped 2 previous similar messages [ 4825.133269] Lustre: server umount lustre-OST0000 complete [ 4833.205808] Lustre: server umount lustre-MDT0000 complete [ 4835.177934] LustreError: 172665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878996 with bad export cookie 15639846187967191929 [ 4835.180591] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.182876] LustreError: 172665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4835.350178] Lustre: server umount lustre-MDT0001 complete [ 4839.674313] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:49:58 (1788878998) [ 4847.458529] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 4852.290135] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4871.632805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4871.751108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4871.765564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4871.767950] Lustre: Skipped 3 previous similar messages [ 4871.815572] Lustre: lustre-MDT0000: new disk, initializing [ 4871.818309] Lustre: Skipped 2 previous similar messages [ 4871.861330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.865665] Lustre: Skipped 2 previous similar messages [ 4871.872119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4871.876103] Lustre: Skipped 2 previous similar messages [ 4874.183518] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4880.537240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4880.600885] Lustre: 178967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4880.772162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4883.205224] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4888.905383] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4892.292575] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4896.460554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4896.649342] Lustre: 180946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4900.071479] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4906.081306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4906.094774] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4911.079074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.079401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.084923] LustreError: Skipped 1 previous similar message [ 4911.090937] Lustre: Skipped 3 previous similar messages [ 4911.094681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.099044] Lustre: Skipped 5 previous similar messages [ 4915.497596] Lustre: server umount lustre-OST0000 complete [ 4923.572848] Lustre: server umount lustre-MDT0000 complete [ 4925.465465] LustreError: 178957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879086 with bad export cookie 15639846187967193014 [ 4925.469747] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.470238] LustreError: 178957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4925.643278] Lustre: server umount lustre-MDT0001 complete [ 4930.727160] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 4932.071758] Key type lgssc unregistered [ 4932.256713] LNet: 182797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4932.261616] LNetError: 182797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4932.272709] LNet: Removed LNI 192.168.204.133@tcp [ 4932.756907] Key type .llcrypt unregistered [ 4932.758849] Key type ._llcrypt unregistered [ 4946.533243] Key type ._llcrypt registered [ 4946.536388] Key type .llcrypt registered [ 4946.600152] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 4947.108453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4947.144404] alg: No test for adler32 (adler32-zlib) [ 4948.041690] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 4948.170575] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 4949.784398] Key type lgssc registered [ 4950.372825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4954.530816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4954.543528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.764832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.766895] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4961.786751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.001088] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4963.856980] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4967.728092] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4970.518378] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4974.091590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4974.276464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.972297] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 4980.458084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4980.642615] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5001.186245] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5001.192204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5001.200367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.209165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.212994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.215061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.225142] Lustre: Skipped 1 previous similar message [ 5005.815612] Lustre: server umount lustre-OST0000 complete [ 5008.353931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5008.359213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.367631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.451735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.452565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.193655] Lustre: server umount lustre-MDT0000 complete [ 5016.432227] LustreError: 184169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879177 with bad export cookie 16811761848684438871 [ 5016.436763] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5016.441116] LustreError: 184169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5016.624883] Lustre: server umount lustre-MDT0001 complete [ 5022.646857] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5024.279831] Key type lgssc unregistered [ 5024.453740] LNet: 187742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5024.457441] LNetError: 187742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5024.470541] LNet: Removed LNI 192.168.204.133@tcp [ 5024.955562] Key type .llcrypt unregistered [ 5024.957423] Key type ._llcrypt unregistered [ 5039.029988] Key type ._llcrypt registered [ 5039.031445] Key type .llcrypt registered [ 5039.087374] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5039.446280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.455385] alg: No test for adler32 (adler32-zlib) [ 5040.388443] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 5040.537886] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5042.160228] Key type lgssc registered [ 5042.735546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5047.134395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5047.148164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.400914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5050.807796] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5055.339831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5055.551866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5057.817416] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5061.731473] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5064.651123] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5068.084626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5068.250234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.607805] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5073.383122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5073.976436] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5083.617765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5083.622714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.640293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.666473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5085.667563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5085.668205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.921383] Lustre: server umount lustre-OST0000 complete [ 5091.934125] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98de204dc000.ost_server_uuid 50 [ 5092.918289] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5103.507404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.510355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.513215] Lustre: Skipped 1 previous similar message [ 5103.516452] Lustre: Skipped 2 previous similar messages [ 5105.572327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5105.576711] Lustre: Skipped 2 previous similar messages [ 5109.667162] Lustre: server umount lustre-MDT0000 complete [ 5110.753882] LustreError: 189133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.764925] LustreError: 189133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5111.498040] LustreError: 189113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879272 with bad export cookie 16269095043965990472 [ 5111.500543] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5111.502582] LustreError: 189113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5111.653309] Lustre: server umount lustre-MDT0001 complete [ 5115.618857] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:54:34 (1788879274) [ 5118.769555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5118.980555] LustreError: 193024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5119.016737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5120.832057] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5124.066481] LustreError: 193024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.072353] LustreError: 193024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5124.415613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.589136] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5126.248172] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5129.445135] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5132.000611] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5135.483676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.672310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.319718] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5141.777856] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5143.850108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5145.426572] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5145.521167] Lustre: lustre-OST0001: new disk, initializing [ 5145.524866] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5145.530720] Lustre: 195862:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5148.840444] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5153.821060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5153.828222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5153.887247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5154.494387] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5158.890279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.892111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5158.895647] Lustre: Skipped 1 previous similar message [ 5158.899610] Lustre: Skipped 1 previous similar message [ 5162.131329] Lustre: server umount lustre-OST0000 complete [ 5164.006129] LustreError: 194843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5164.013418] LustreError: 194843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5165.576288] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5165.686317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5167.860381] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5167.961555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5169.126111] LustreError: 196534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5169.133576] LustreError: 196534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5179.361654] LustreError: 194844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5179.364697] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.365297] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5179.365302] Lustre: Skipped 1 previous similar message [ 5179.371030] LustreError: 194844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5179.377826] Lustre: Skipped 1 previous similar message [ 5179.391333] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5185.388067] Lustre: server umount lustre-OST0001 complete [ 5189.602884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5189.608993] Lustre: Skipped 1 previous similar message [ 5199.841530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5199.846698] Lustre: Skipped 8 previous similar messages [ 5201.376127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5201.452354] Lustre: server umount lustre-MDT0000 complete [ 5203.204605] LustreError: 193005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879364 with bad export cookie 16269095043965991529 [ 5203.206921] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5203.210559] LustreError: 193005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5203.341691] Lustre: server umount lustre-MDT0001 complete [ 5214.840836] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:56:13 (1788879373) [ 5218.055253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.199436] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5218.218453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5218.318684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.322676] Lustre: Skipped 1 previous similar message [ 5220.250367] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5224.298959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.355503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5224.378725] Lustre: 199291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5226.525318] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5230.186513] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5233.140759] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5236.604714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5236.705106] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5239.618213] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5243.009164] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5246.287237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5246.330965] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5246.828223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5246.844741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5249.204697] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5252.374962] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5257.186819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5257.190674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5257.191759] LustreError: Skipped 2 previous similar messages [ 5257.197707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5257.202071] Lustre: Skipped 1 previous similar message [ 5259.617954] Lustre: server umount lustre-OST0000 complete [ 5262.311378] LustreError: 201136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.319544] LustreError: 201136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5272.547662] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5272.550785] LustreError: Skipped 1 previous similar message [ 5277.277783] Lustre: server umount lustre-OST0001 complete [ 5280.737098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5285.113490] Lustre: server umount lustre-MDT0000 complete [ 5286.779811] LustreError: 199284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879447 with bad export cookie 16269095043965992915 [ 5286.780118] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.785386] LustreError: 199284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5286.953789] Lustre: server umount lustre-MDT0001 complete [ 5297.982580] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:57:37 (1788879457) [ 5305.036647] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5309.424545] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5325.769660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.885521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5325.905764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5325.954585] Lustre: lustre-MDT0000: new disk, initializing [ 5326.002521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.011192] Lustre: Skipped 3 previous similar messages [ 5326.024836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5328.147785] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5333.662872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.727465] Lustre: 207350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5333.758208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5333.760926] Lustre: Skipped 1 previous similar message [ 5333.803135] Lustre: lustre-MDT0001: new disk, initializing [ 5333.876454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5333.879854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5335.708884] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5340.859078] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5343.475309] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5346.988458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.116617] Lustre: lustre-OST0000: new disk, initializing [ 5347.119946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5347.125313] Lustre: 209319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5349.821595] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5354.793200] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5356.565092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5356.570911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5356.614759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5356.758640] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5356.863814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5358.684296] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5358.801235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5361.635736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5361.636933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5361.640834] LustreError: Skipped 1 previous similar message [ 5361.648450] Lustre: Skipped 5 previous similar messages [ 5361.649883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5361.657347] Lustre: Skipped 7 previous similar messages [ 5365.802878] Lustre: server umount lustre-OST0000 complete [ 5373.756037] Lustre: server umount lustre-MDT0000 complete [ 5375.324229] LustreError: 208146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879536 with bad export cookie 16269095043965994588 [ 5375.326466] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5375.329611] LustreError: 208146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5375.469432] Lustre: server umount lustre-MDT0001 complete [ 5379.335790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.549801] LustreError: 211532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.559754] LustreError: 211532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5381.500578] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5385.432182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5387.406703] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5390.595088] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5392.886469] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5395.760055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.926466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5395.929606] Lustre: Skipped 4 previous similar messages [ 5398.636490] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5401.908713] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5403.979350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5406.144163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5408.124920] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5408.216985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5410.281722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5410.287706] LustreError: Skipped 1 previous similar message [ 5411.311160] Lustre: server umount lustre-OST0000 complete [ 5414.120774] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5414.237640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5416.197875] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5440.843441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5443.498532] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5446.712511] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5447.404962] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5455.010122] Lustre: server umount lustre-OST0000 complete [ 5457.377136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5457.383393] Lustre: Skipped 5 previous similar messages [ 5462.894116] Lustre: server umount lustre-MDT0000 complete [ 5464.657051] LustreError: 211512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879625 with bad export cookie 16269095043965995470 [ 5464.657528] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.662197] LustreError: 211512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5464.823436] Lustre: server umount lustre-MDT0001 complete [ 5468.878280] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:00:28 (1788879628) [ 5472.045660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5472.301209] LustreError: 217519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5472.311522] LustreError: 217519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5474.391251] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5478.398887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5480.638442] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5484.029626] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5486.356377] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5489.238548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.922330] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5495.141563] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5497.189062] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5498.345789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5500.316415] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5504.545695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 5507.965604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5508.081473] Lustre: lustre-OST0001: new disk, initializing [ 5508.083974] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5508.087888] Lustre: 220935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5511.164532] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5514.770710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5514.777522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5514.832809] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5516.116047] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5518.090972] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5518.178628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5520.110483] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5520.201725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5524.967618] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5524.969189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5524.972925] LustreError: Skipped 1 previous similar message [ 5524.975544] Lustre: Skipped 13 previous similar messages [ 5527.241617] Lustre: server umount lustre-OST0001 complete [ 5529.836439] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5529.917696] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5531.673833] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5531.764133] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5556.180138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.286989] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.289838] Lustre: Skipped 5 previous similar messages [ 5558.657567] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5561.324498] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5561.332257] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5561.333893] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5561.345266] Lustre: Skipped 1 previous similar message [ 5561.656348] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5562.259884] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5569.087710] Lustre: server umount lustre-OST0001 complete [ 5585.892669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.900304] Lustre: Skipped 6 previous similar messages [ 5588.379738] Lustre: server umount lustre-OST0000 complete [ 5604.320294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5604.404456] Lustre: server umount lustre-MDT0000 complete [ 5605.827216] LustreError: 217544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879766 with bad export cookie 16269095043965996408 [ 5605.830743] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5605.831807] LustreError: 217544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5620.192154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5620.307479] Lustre: server umount lustre-MDT0001 complete [ 5629.738773] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:03:09 (1788879789) [ 5632.404762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.553182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5632.568267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5634.454208] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5638.230479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5638.297895] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5638.310940] Lustre: 226171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5640.530128] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5643.836133] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5646.274365] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5649.505899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.616608] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5652.226622] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5655.463323] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5665.124633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5665.169844] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5667.943730] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5671.326691] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5671.980472] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5673.903676] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5673.994425] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5675.685918] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5675.785617] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5677.972363] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98de0566a000.ost_server_uuid 50 [ 5678.588635] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98de0566a000.ost_server_uuid in FULL state after 0 sec [ 5678.966584] Lustre: Permanently deactivating lustre-OST0001 [ 5678.972268] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5679.869265] Lustre: Permanently reactivating lustre-OST0001 [ 5681.640393] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5681.643737] LustreError: Skipped 6 previous similar messages [ 5687.041261] Lustre: server umount lustre-OST0001 complete [ 5691.875362] LustreError: 228698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.881331] LustreError: 228698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 5695.252584] Lustre: server umount lustre-OST0000 complete [ 5703.259742] Lustre: server umount lustre-MDT0000 complete [ 5704.597865] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879865 with bad export cookie 16269095043965997682 [ 5704.602652] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.603831] LustreError: 226914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5704.727722] Lustre: server umount lustre-MDT0001 complete [ 5713.367291] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:04:32 (1788879872) [ 5717.428738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5717.518500] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5717.529929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5717.532941] Lustre: Skipped 5 previous similar messages [ 5719.310332] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5722.871796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.929855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5722.946179] Lustre: 232857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5724.683527] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5727.545082] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5730.388491] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5733.549520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5733.670329] Lustre: Permanently deactivating lustre-OST0000 [ 5733.680777] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5736.734397] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5740.205125] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5742.589731] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5743.119540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.680037] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5748.686208] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5749.808446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5754.437153] Lustre: Permanently reactivating lustre-OST0000 [ 5754.443021] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5754.446024] Lustre: Skipped 2 previous similar messages [ 5759.497869] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5759.501207] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5759.508357] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5759.530771] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5759.541282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5765.830838] Lustre: server umount lustre-OST0001 complete [ 5773.399752] Lustre: server umount lustre-OST0000 complete [ 5781.805432] Lustre: server umount lustre-MDT0000 complete [ 5783.441912] LustreError: 233595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879944 with bad export cookie 16269095043965999320 [ 5783.446479] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.446598] LustreError: 233595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5783.586108] Lustre: server umount lustre-MDT0001 complete [ 5787.910621] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 5789.217763] Key type lgssc unregistered [ 5789.366350] LNet: 237708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5789.371069] LNetError: 237708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5789.379547] LNet: Removed LNI 192.168.204.133@tcp [ 5789.822467] Key type .llcrypt unregistered [ 5789.824155] Key type ._llcrypt unregistered [ 5800.295469] Key type ._llcrypt registered [ 5800.297177] Key type .llcrypt registered [ 5800.347653] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5800.767257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5800.807263] alg: No test for adler32 (adler32-zlib) [ 5801.703038] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 5801.816435] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 5803.408252] Key type lgssc registered [ 5803.916578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5808.250170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5808.259346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.488322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.328635] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5815.801584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5815.884307] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5815.899810] Lustre: 239137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5815.905749] Lustre: Permanently deactivating lustre-MDT0001 [ 5815.913686] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5816.098671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5818.584622] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5819.062486] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5819.065058] LustreError: 239180:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5819.071781] LustreError: 239180:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5819.077545] LustreError: 239180:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 5821.757329] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5824.108177] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5827.135447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5827.297501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.660291] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5832.723184] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5834.468147] LustreError: 240992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.474808] LustreError: 240992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5834.478708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5835.638834] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.749241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5838.228667] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5840.875859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5840.878166] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5841.341137] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5843.009587] Lustre: Permanently reactivating lustre-MDT0001 [ 5843.018612] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5843.021347] Lustre: Skipped 2 previous similar messages [ 5849.192696] Lustre: lustre-MDT0001: Client db07c03e-5433-416d-ad49-41ee8507695a (at 192.168.204.33@tcp) reconnecting [ 5851.459296] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.061217] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5854.111585] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5854.196650] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5855.928492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5856.021696] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5856.988745] Lustre: Permanently deactivating lustre-MDT0001 [ 5856.995901] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5856.998956] Lustre: Skipped 2 previous similar messages [ 5866.514079] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5871.586205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5871.589297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5871.591130] Lustre: Skipped 3 previous similar messages [ 5871.594764] Lustre: Skipped 3 previous similar messages [ 5876.706939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5876.711859] Lustre: Skipped 3 previous similar messages [ 5877.596896] Lustre: server umount lustre-MDT0000 complete [ 5879.142401] LustreError: 239888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880039 with bad export cookie 17839396380017694329 [ 5879.145120] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.147461] LustreError: 239888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5879.279667] Lustre: server umount lustre-MDT0001 complete [ 5886.914111] Lustre: server umount lustre-OST0000 complete [ 5891.160854] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:07:30 (1788880050) [ 5895.493647] Lustre: server umount lustre-OST0001 complete [ 5898.242696] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 5902.334899] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 5916.898417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.010497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5917.015859] Lustre: Skipped 2 previous similar messages [ 5917.028537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5917.073915] Lustre: lustre-MDT0000: new disk, initializing [ 5917.109994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5917.117910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5918.728561] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5923.779190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.830140] Lustre: 247491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.859536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5923.862064] Lustre: Skipped 1 previous similar message [ 5923.902459] Lustre: lustre-MDT0001: new disk, initializing [ 5923.964714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5923.968794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5925.603035] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5929.964665] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5932.416317] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5935.032949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5935.152214] Lustre: lustre-OST0000: new disk, initializing [ 5935.154315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5935.158636] Lustre: 249462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5935.197497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5935.201242] Lustre: Skipped 1 previous similar message [ 5937.514328] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5942.037604] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5942.784609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5942.788540] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5942.828047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5948.134632] LustreError: 247528:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5952.623701] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.709612] Lustre: lustre-OST0001: new disk, initializing [ 5952.712237] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5952.715570] Lustre: 250578:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5952.756950] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5954.951257] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 5958.166294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5958.172868] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5958.224594] LustreError: 247528:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5958.231441] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5958.266887] LustreError: 247500:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5959.662646] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5960.351171] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5968.362181] LustreError: 247500:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5973.105405] Lustre: lustre-OST0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 5973.474490] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5973.475947] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.477613] LustreError: Skipped 1 previous similar message [ 5975.299413] Lustre: server umount lustre-OST0001 complete [ 5978.596862] LustreError: 249446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.610789] LustreError: 249446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5983.716951] LustreError: 249446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.725263] LustreError: 249446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5988.838926] LustreError: 250181:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.839857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5988.840241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5988.840253] Lustre: Skipped 1 previous similar message [ 5988.841621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5988.841626] Lustre: Skipped 2 previous similar messages [ 5988.846959] LustreError: 250181:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5988.854607] LustreError: Skipped 1 previous similar message [ 5993.955153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5993.956018] LustreError: 249445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.958260] Lustre: Skipped 2 previous similar messages [ 5993.965824] LustreError: 249445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5994.069903] Lustre: server umount lustre-OST0000 complete [ 5999.074469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.080184] Lustre: Skipped 2 previous similar messages [ 6001.818467] Lustre: server umount lustre-MDT0000 complete [ 6003.219216] LustreError: 250577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880164 with bad export cookie 17839396380017696884 [ 6003.223714] LustreError: 250577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6003.224083] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6009.400210] Lustre: server umount lustre-MDT0001 complete [ 6013.179302] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6014.260841] Key type lgssc unregistered [ 6014.398423] LNet: 252610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.401684] LNetError: 252610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6014.414333] LNet: Removed LNI 192.168.204.133@tcp [ 6014.736668] Key type .llcrypt unregistered [ 6014.737856] Key type ._llcrypt unregistered [ 6029.482673] Key type ._llcrypt registered [ 6029.483881] Key type .llcrypt registered [ 6029.523935] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6029.849472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6029.856509] alg: No test for adler32 (adler32-zlib) [ 6030.717773] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6030.816240] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6032.400163] Key type lgssc registered [ 6032.802461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6035.549737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6035.556055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.642509] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6036.650823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6036.713917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.042151] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6040.910136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.942183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6040.952770] Lustre: 254790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6041.051531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6042.451945] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6044.893474] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6046.737398] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6049.079887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6049.158415] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6049.230987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.322483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6051.362810] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6053.893370] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6096.355274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.355430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6096.359551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6096.362172] LustreError: Skipped 1 previous similar message [ 6098.718045] Lustre: server umount lustre-OST0000 complete [ 6100.299214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6104.792905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6104.930679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.198769] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6109.893814] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6116.195855] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.199390] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6116.199551] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6116.205095] Lustre: Skipped 1 previous similar message [ 6147.041310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.041482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.043933] LustreError: Skipped 1 previous similar message [ 6147.047866] Lustre: Skipped 1 previous similar message [ 6147.049934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.053184] Lustre: Skipped 2 previous similar messages [ 6149.403993] Lustre: server umount lustre-OST0000 complete [ 6152.162365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.163355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6152.168860] Lustre: Skipped 1 previous similar message [ 6156.893085] Lustre: server umount lustre-MDT0000 complete [ 6158.330723] LustreError: 259250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880319 with bad export cookie 5520610682119736502 [ 6158.334400] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6158.336972] LustreError: 259250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6158.454454] Lustre: server umount lustre-MDT0001 complete [ 6162.524985] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6163.637640] Key type lgssc unregistered [ 6163.775534] LNet: 260202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6163.779203] LNetError: 260202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6163.786323] LNet: Removed LNI 192.168.204.133@tcp [ 6164.124682] Key type .llcrypt unregistered [ 6164.125975] Key type ._llcrypt unregistered [ 6173.368091] Key type ._llcrypt registered [ 6173.369033] Key type .llcrypt registered [ 6173.409035] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6173.772943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6173.797558] alg: No test for adler32 (adler32-zlib) [ 6174.665181] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6174.766867] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6176.352145] Key type lgssc registered [ 6176.773877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6179.701469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6179.709849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.879996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6182.527310] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6185.888462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.047838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6187.648224] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6190.363796] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.418213] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6195.045783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6195.197932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6197.339796] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6200.018286] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6205.481640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6215.650280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6215.650823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.652259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6215.658328] Lustre: Skipped 1 previous similar message [ 6219.346393] Lustre: server umount lustre-OST0000 complete [ 6220.769353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.769967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6220.774535] Lustre: Skipped 1 previous similar message [ 6220.776077] Lustre: Skipped 1 previous similar message [ 6225.889164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6225.891448] Lustre: Skipped 2 previous similar messages [ 6226.787490] Lustre: server umount lustre-MDT0000 complete [ 6228.022409] LustreError: 261573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880388 with bad export cookie 8260857677245641114 [ 6228.025209] LustreError: 261573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6228.025621] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6228.141322] Lustre: server umount lustre-MDT0001 complete [ 6231.845475] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6232.898411] Key type lgssc unregistered [ 6233.026093] LNet: 265894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6233.029306] LNetError: 265894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6233.036662] LNet: Removed LNI 192.168.204.133@tcp [ 6233.382244] Key type .llcrypt unregistered [ 6233.383545] Key type ._llcrypt unregistered [ 6240.511806] Key type ._llcrypt registered [ 6240.513024] Key type .llcrypt registered [ 6240.553433] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6240.873266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6240.905077] alg: No test for adler32 (adler32-zlib) [ 6241.765968] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6241.857608] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6243.440146] Key type lgssc registered [ 6243.819597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6246.465124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6246.472038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.616208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6249.042725] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6251.991890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.113744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.479828] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6255.785989] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6257.606195] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6260.015382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6260.132455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6262.040286] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6264.558213] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6272.423074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6281.697129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6281.700688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.706706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.724085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.725209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.726936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.894150] Lustre: server umount lustre-OST0000 complete [ 6287.841860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6287.842344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6287.842656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6287.842661] Lustre: Skipped 1 previous similar message [ 6287.847562] Lustre: Skipped 1 previous similar message [ 6292.334779] Lustre: server umount lustre-MDT0000 complete [ 6292.961025] LustreError: 267088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.967176] LustreError: 267088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6293.847105] LustreError: 267070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880454 with bad export cookie 15620824888297115678 [ 6293.850344] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6293.853949] LustreError: 267070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6293.993751] Lustre: server umount lustre-MDT0001 complete [ 6298.052184] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6299.153481] Key type lgssc unregistered [ 6299.292442] LNet: 270785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6299.294576] LNetError: 270785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6299.305348] LNet: Removed LNI 192.168.204.133@tcp [ 6299.630289] Key type .llcrypt unregistered [ 6299.631351] Key type ._llcrypt unregistered [ 6309.117210] Key type ._llcrypt registered [ 6309.118500] Key type .llcrypt registered [ 6309.158791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6309.556683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6309.580850] alg: No test for adler32 (adler32-zlib) [ 6310.433588] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6310.522235] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6312.104152] Key type lgssc registered [ 6312.498600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6315.216384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6315.222369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.381073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.834203] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6321.062529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.224409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6322.739714] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6325.515917] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6327.665713] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6330.366446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.510394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.552253] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6335.054891] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6342.821069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6357.472830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6357.476170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.482266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6358.499429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.501635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6358.501694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.273941] Lustre: server umount lustre-OST0000 complete [ 6362.080917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6362.084290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.090055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6362.092094] Lustre: Skipped 1 previous similar message [ 6367.707853] Lustre: server umount lustre-MDT0000 complete [ 6368.738301] LustreError: 272173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.744273] LustreError: 272173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6369.036346] LustreError: 272155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880529 with bad export cookie 16130550522652291392 [ 6369.039064] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6369.041077] LustreError: 272155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6369.150863] Lustre: server umount lustre-MDT0001 complete [ 6373.024837] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6374.130906] Key type lgssc unregistered [ 6374.255428] LNet: 276495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6374.258852] LNetError: 276495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6374.266329] LNet: Removed LNI 192.168.204.133@tcp [ 6374.609562] Key type .llcrypt unregistered [ 6374.610876] Key type ._llcrypt unregistered [ 6381.713136] Key type ._llcrypt registered [ 6381.714092] Key type .llcrypt registered [ 6381.746041] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6382.060671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6382.110429] alg: No test for adler32 (adler32-zlib) [ 6382.973772] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6383.063343] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6384.640166] Key type lgssc registered [ 6385.025599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6387.980480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6387.988946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.164218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6390.938946] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6394.027691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6394.174137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6395.587212] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6398.010361] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6399.913824] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6402.256562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.386162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6404.408384] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6407.078899] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6414.696118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6424.033097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6424.036586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6424.040779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.058295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6425.060058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.065296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6428.050525] Lustre: server umount lustre-OST0000 complete [ 6430.177423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6430.178172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.179085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6430.179088] Lustre: Skipped 1 previous similar message [ 6430.187212] Lustre: Skipped 1 previous similar message [ 6435.297265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.298936] Lustre: Skipped 2 previous similar messages [ 6435.477933] Lustre: server umount lustre-MDT0000 complete [ 6437.162582] LustreError: 277670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880598 with bad export cookie 1709160069789439259 [ 6437.164541] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6437.168768] LustreError: 277670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6437.294449] Lustre: server umount lustre-MDT0001 complete [ 6441.482598] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6442.658740] Key type lgssc unregistered [ 6442.820549] LNet: 281422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6442.824347] LNetError: 281422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6442.836903] LNet: Removed LNI 192.168.204.133@tcp [ 6443.234915] Key type .llcrypt unregistered [ 6443.236351] Key type ._llcrypt unregistered [ 6447.349087] Key type ._llcrypt registered [ 6447.350392] Key type .llcrypt registered [ 6447.394795] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6453.423860] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6459.755067] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6460.055251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6460.068362] alg: No test for adler32 (adler32-zlib) [ 6460.928619] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6461.018530] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6462.600124] Key type lgssc registered [ 6462.972392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6474.593149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6474.599431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.683764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6475.694833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6475.733526] Lustre: lustre-MDT0000: new disk, initializing [ 6475.758364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6475.764887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6477.258329] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6481.781260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.827962] Lustre: 284941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6481.852870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6481.855181] Lustre: Skipped 1 previous similar message [ 6481.894589] Lustre: lustre-MDT0001: new disk, initializing [ 6481.928162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6481.942254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6481.946489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6483.457968] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6487.465558] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6489.743222] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6492.278424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6492.411829] Lustre: lustre-OST0000: new disk, initializing [ 6492.412663] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 6492.416136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6492.421188] Lustre: 286911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6492.463025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.185041] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6497.787363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6497.791570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6497.806561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 6500.628685] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6502.288369] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6502.371257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6504.039995] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6504.128852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6508.002039] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6508.008651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.015342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6508.516959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6508.519500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.634585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6513.636594] Lustre: Skipped 2 previous similar messages [ 6518.754733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.756360] Lustre: Skipped 1 previous similar message [ 6519.264083] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6519.316410] Lustre: server umount lustre-OST0000 complete [ 6522.848826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6522.851341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.854323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.805395] Lustre: server umount lustre-MDT0000 complete [ 6529.074346] LustreError: 285738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880689 with bad export cookie 13363914798300215026 [ 6529.078050] LustreError: 285738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6529.079278] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.198611] Lustre: server umount lustre-MDT0001 complete [ 6532.676834] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:18:12 (1788880692) [ 6535.028201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.200743] LustreError: 289322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6535.207261] LustreError: 289322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6535.223687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.591161] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6539.558323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6539.673317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6541.993267] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6544.611134] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6546.385462] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6548.567045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6548.677153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6550.592198] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6552.936182] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6570.465241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6570.467620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6570.471411] Lustre: Skipped 1 previous similar message [ 6570.473228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6570.474821] Lustre: Skipped 2 previous similar messages [ 6572.503300] Lustre: server umount lustre-OST0000 complete [ 6575.584679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6575.586900] LustreError: Skipped 1 previous similar message [ 6575.588095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6575.591299] Lustre: Skipped 1 previous similar message [ 6586.848839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6586.850700] Lustre: Skipped 8 previous similar messages [ 6588.384081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6588.437880] Lustre: server umount lustre-MDT0000 complete [ 6589.617989] LustreError: 289304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880750 with bad export cookie 13363914798300215894 [ 6589.620722] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.621184] LustreError: 289304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6589.734896] Lustre: server umount lustre-MDT0001 complete [ 6592.842929] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6593.763778] Key type lgssc unregistered [ 6593.884363] LNet: 292878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6593.886337] LNetError: 292878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6593.893264] LNet: Removed LNI 192.168.204.133@tcp [ 6594.161409] Key type .llcrypt unregistered [ 6594.162341] Key type ._llcrypt unregistered [ 6602.680015] Key type ._llcrypt registered [ 6602.681402] Key type .llcrypt registered [ 6602.718562] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6608.781198] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6609.199100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6609.206696] alg: No test for adler32 (adler32-zlib) [ 6610.074196] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6610.159398] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6611.736188] Key type lgssc registered [ 6612.069263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6626.491938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6626.496614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6627.578189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6627.587249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6627.622882] Lustre: lustre-MDT0000: new disk, initializing [ 6627.647164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6627.654770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6629.106645] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6633.427039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.470479] Lustre: 296670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6633.501789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6633.504656] Lustre: Skipped 1 previous similar message [ 6633.539984] Lustre: lustre-MDT0001: new disk, initializing [ 6633.575318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6633.588873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6633.592243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6635.025300] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6638.701598] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6640.525837] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6642.750588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6642.866112] Lustre: lustre-OST2710: new disk, initializing [ 6642.867756] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6642.870138] Lustre: 298638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6642.899937] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6644.678646] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6648.313707] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.337508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6649.340675] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6649.375426] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6650.532703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.595674] Lustre: lustre-OST03e8: new disk, initializing [ 6650.597247] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6650.600147] Lustre: 299699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6650.634673] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6652.523371] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6654.886027] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6657.026735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6657.031576] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6657.065598] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6657.549817] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6657.614824] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6658.944300] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6659.026305] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6660.352350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6660.425456] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6661.750760] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6661.813469] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6664.160779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6664.162858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6664.167709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.233519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.235393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6668.888686] Lustre: server umount lustre-MDT0000 complete [ 6670.089968] LustreError: 296663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880830 with bad export cookie 8567579341654691121 [ 6670.093934] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6670.094219] LustreError: 296663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6670.200533] Lustre: server umount lustre-MDT0001 complete [ 6681.608366] Lustre: server umount lustre-OST2710 complete [ 6693.147801] Lustre: server umount lustre-OST03e8 complete [ 6698.890349] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 6701.947350] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6713.278830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.357189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6713.368214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6713.403512] Lustre: lustre-MDT0000: new disk, initializing [ 6713.422104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.426832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6714.734899] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6718.889291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6718.925653] Lustre: 304938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6719.011374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6720.257303] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6723.864188] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6725.555560] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6727.685363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6727.799431] Lustre: lustre-OST0000: new disk, initializing [ 6727.800595] Lustre: Skipped 1 previous similar message [ 6727.801910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6727.803544] Lustre: Skipped 2 previous similar messages [ 6727.805211] Lustre: 306909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6727.839253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6727.841102] Lustre: Skipped 1 previous similar message [ 6729.755019] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6733.506500] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6734.868564] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6737.912605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6737.916083] Lustre: Skipped 1 previous similar message [ 6737.918251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6737.956306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6738.018421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6739.419202] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6739.492250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6743.010596] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6743.010644] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6743.012563] LustreError: Skipped 1 previous similar message [ 6743.015673] Lustre: Skipped 2 previous similar messages [ 6743.016189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6743.019510] Lustre: Skipped 4 previous similar messages [ 6746.251350] Lustre: server umount lustre-OST0000 complete [ 6748.129466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6748.134888] Lustre: Skipped 1 previous similar message [ 6748.136920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6748.138610] Lustre: Skipped 1 previous similar message [ 6753.248872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6753.250595] Lustre: Skipped 1 previous similar message [ 6753.560794] Lustre: server umount lustre-MDT0000 complete [ 6754.764379] LustreError: 304931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880915 with bad export cookie 8567579341654692780 [ 6754.767720] LustreError: 304931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6754.769310] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6754.885153] Lustre: server umount lustre-MDT0001 complete [ 6757.368677] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:21:56 (1788880916) [ 6757.853320] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6758.403335] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:21:57 (1788880917) [ 6759.366320] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6767.736047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.822583] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6767.829670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6767.883654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.146635] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6771.846092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6771.880244] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6771.889897] Lustre: 310784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6773.294205] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6789.090882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.160639] Lustre: Denying initial registration attempt from nid 192.168.204.133@tcp, specified as failover [ 6789.163464] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6789.166162] LustreError: 312236:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 6789.167982] LustreError: 312236:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 6789.169554] LustreError: 312236:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 6789.192489] Lustre: server umount lustre-OST0000 complete [ 6789.193643] LustreError: 312236:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6792.672795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6792.675146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.678760] Lustre: Skipped 1 previous similar message [ 6792.680389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.330424] Lustre: server umount lustre-MDT0000 complete [ 6798.555483] LustreError: 310775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880959 with bad export cookie 8567579341654693648 [ 6798.558387] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6798.558887] LustreError: 310775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6798.673911] Lustre: server umount lustre-MDT0001 complete [ 6802.153300] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6803.164840] Key type lgssc unregistered [ 6803.300396] LNet: 313355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6803.303180] LNetError: 313355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6803.312322] LNet: Removed LNI 192.168.204.133@tcp [ 6803.606334] Key type .llcrypt unregistered [ 6803.607947] Key type ._llcrypt unregistered [ 6806.909936] Key type ._llcrypt registered [ 6806.911358] Key type .llcrypt registered [ 6806.945483] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6807.228283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6807.234397] alg: No test for adler32 (adler32-zlib) [ 6808.075469] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6808.165564] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6809.744200] Key type lgssc registered [ 6810.092293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6833.811793] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6836.939474] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6836.945704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.046241] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6838.053272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6838.110795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.514841] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6842.390179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.426667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6842.439223] Lustre: 316095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6842.540423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6843.903784] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6846.362043] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6848.200501] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6863.587332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.664699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6863.739696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.620995] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6867.920711] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6868.606194] Lustre: server umount lustre-OST0000 complete [ 6873.569116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.572078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.576352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.689137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.691983] Lustre: Skipped 2 previous similar messages [ 6883.809152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6883.810824] Lustre: Skipped 1 previous similar message [ 6884.320074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6884.366610] Lustre: server umount lustre-MDT0000 complete [ 6885.638300] LustreError: 316833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881046 with bad export cookie 406816117717967160 [ 6885.641487] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6885.644058] LustreError: 316833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6885.761693] Lustre: server umount lustre-MDT0001 complete [ 6889.381150] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 6890.541555] Key type lgssc unregistered [ 6890.685549] LNet: 319658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6890.688586] LNetError: 319658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6890.696414] LNet: Removed LNI 192.168.204.133@tcp [ 6891.031673] Key type .llcrypt unregistered [ 6891.033700] Key type ._llcrypt unregistered [ 6900.401253] Key type ._llcrypt registered [ 6900.402140] Key type .llcrypt registered [ 6900.438173] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 6900.835626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6900.840989] alg: No test for adler32 (adler32-zlib) [ 6901.684767] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 6901.766821] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 6903.344110] Key type lgssc registered [ 6903.675600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6906.044466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6906.051327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.179869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6908.401054] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6911.009042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6911.125773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6912.441472] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6914.811331] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6916.581312] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6920.867275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.970274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6922.754213] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6925.058420] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6940.641892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6940.643711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6940.645983] Lustre: Skipped 2 previous similar messages [ 6940.647653] Lustre: Skipped 1 previous similar message [ 6941.793909] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 6941.796886] Lustre: Skipped 1 previous similar message [ 6944.664934] Lustre: server umount lustre-MDT0000 complete [ 6945.761343] LustreError: 321774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.766985] LustreError: 321774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6945.842119] LustreError: 321027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881106 with bad export cookie 9601857127055261321 [ 6945.845493] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6945.847380] LustreError: 321027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6946.906537] LustreError: 321042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.912540] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 6950.880758] LustreError: 322516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6950.880950] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.887708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6956.001105] LustreError: 321043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6956.001779] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6956.004940] LustreError: 321043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6956.006025] Lustre: Skipped 1 previous similar message [ 6960.032702] LustreError: 322516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6960.036208] LustreError: 322516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6960.096095] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6960.172943] Lustre: server umount lustre-MDT0001 complete [ 6962.192340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6964.997830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.272687] LustreError: 320357:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008c59640c x1875777748493568/t0(0) o250->MGC192.168.204.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6970.382163] LustreError: 322877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6970.406740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.433229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6971.862820] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6974.665316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6974.796927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6981.206853] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 6983.609127] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6985.191807] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6989.238832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 6991.163772] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.785433] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6995.426410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6995.429168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.430209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6995.435615] Lustre: Skipped 1 previous similar message [ 6995.439174] Lustre: Skipped 3 previous similar messages [ 6999.198539] Lustre: server umount lustre-OST0000 complete [ 7000.545622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7000.545689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.551493] Lustre: Skipped 1 previous similar message [ 7006.614194] Lustre: server umount lustre-MDT0000 complete [ 7007.865119] LustreError: 321027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881168 with bad export cookie 9601857127055268447 [ 7007.869195] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.870710] LustreError: 321027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7007.989592] Lustre: server umount lustre-MDT0001 complete [ 7011.689393] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7012.767182] Key type lgssc unregistered [ 7012.906353] LNet: 327327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7012.909187] LNetError: 327327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7012.915305] LNet: Removed LNI 192.168.204.133@tcp [ 7013.225131] Key type .llcrypt unregistered [ 7013.226264] Key type ._llcrypt unregistered [ 7022.451323] Key type ._llcrypt registered [ 7022.452949] Key type .llcrypt registered [ 7022.496545] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7022.969552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7023.001685] alg: No test for adler32 (adler32-zlib) [ 7023.889168] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 7023.997864] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7025.584168] Key type lgssc registered [ 7025.987452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7028.624887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7028.631051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.793537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7031.157144] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7034.000488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.138643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7035.532827] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7037.918148] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7039.753535] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7043.810098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.814552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.816345] Lustre: Skipped 1 previous similar message [ 7049.185317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7049.188446] Lustre: Skipped 1 previous similar message [ 7054.305051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7054.307292] Lustre: Skipped 1 previous similar message [ 7055.328092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7055.385832] Lustre: server umount lustre-MDT0000 complete [ 7056.601243] LustreError: 328715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881217 with bad export cookie 7381710038622350181 [ 7056.605418] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7056.717834] Lustre: server umount lustre-MDT0001 complete [ 7061.398916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7061.499118] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7061.510929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7061.583911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.910741] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7065.818618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.855870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7065.867931] Lustre: 331320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7065.965068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7067.288854] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7069.656741] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7071.570437] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7076.258844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7076.330315] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7076.333316] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7076.338083] LustreError: 333198:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 7076.341384] LustreError: 333198:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7076.344855] LustreError: 333198:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7076.373868] Lustre: server umount lustre-OST0000 complete [ 7076.375127] LustreError: 333198:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7081.186869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.254600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7081.318546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.366773] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7085.588579] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7087.927211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.956045] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7087.978720] Lustre: lustre-OST0001: new disk, initializing [ 7087.980389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7087.983338] Lustre: 334728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7088.307228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7088.313782] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7088.329453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7088.357798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7089.748168] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7092.097302] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7093.731340] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7093.734476] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.738203] Lustre: Skipped 1 previous similar message [ 7093.740708] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7093.742759] Lustre: Skipped 1 previous similar message [ 7095.108695] Lustre: server umount lustre-OST0001 complete [ 7098.853803] LustreError: 333665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7098.854363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7098.854468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.854476] Lustre: Skipped 1 previous similar message [ 7098.855450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7098.858750] LustreError: 333665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7098.860785] LustreError: Skipped 2 previous similar messages [ 7102.478631] Lustre: server umount lustre-OST0000 complete [ 7103.969335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.972102] Lustre: Skipped 2 previous similar messages [ 7109.089638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7109.092768] Lustre: Skipped 4 previous similar messages [ 7118.304085] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7118.367452] Lustre: server umount lustre-MDT0000 complete [ 7119.622100] LustreError: 331310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881280 with bad export cookie 7381710038622350720 [ 7119.625263] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7119.627746] LustreError: 331310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7119.748611] Lustre: server umount lustre-MDT0001 complete [ 7123.408387] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7124.446898] Key type lgssc unregistered [ 7124.576330] LNet: 336689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7124.578897] LNetError: 336689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7124.592383] LNet: Removed LNI 192.168.204.133@tcp [ 7124.979558] Key type .llcrypt unregistered [ 7124.981664] Key type ._llcrypt unregistered [ 7185.771440] Key type ._llcrypt registered [ 7185.774541] Key type .llcrypt registered [ 7186.198488] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7224.918476] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7227.812696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7228.616101] alg: No test for adler32 (adler32-zlib) [ 7230.026215] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 7230.423739] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7232.152969] Key type lgssc registered [ 7234.257898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7269.604154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7269.620698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.833840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7270.866293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7270.956732] Lustre: lustre-MDT0000: new disk, initializing [ 7271.055250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7271.110556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7274.693921] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7283.339416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7283.439701] Lustre: 341622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7283.476715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7283.478835] Lustre: Skipped 1 previous similar message [ 7283.559392] Lustre: lustre-MDT0001: new disk, initializing [ 7283.660465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7283.695374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7283.707885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7287.478492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7295.450235] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7300.191466] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7306.289074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.568297] Lustre: lustre-OST0000: new disk, initializing [ 7306.576779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7306.592263] Lustre: 343592:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7306.674426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7312.403656] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7315.999096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7316.009865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7316.092288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7321.076954] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7325.432100] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7325.630682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7330.027517] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7330.247604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7336.420337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7336.429726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7336.429867] Lustre: Skipped 1 previous similar message [ 7336.430331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7336.452244] Lustre: Skipped 1 previous similar message [ 7338.231237] Lustre: server umount lustre-OST0000 complete [ 7341.539565] LustreError: 339841:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a1d1826f x1875778093486976/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7341.540108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7341.595237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7341.605304] Lustre: Skipped 1 previous similar message [ 7346.660069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7346.668522] Lustre: Skipped 1 previous similar message [ 7347.812462] Lustre: server umount lustre-MDT0000 complete [ 7351.019565] LustreError: 341614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881511 with bad export cookie 8814514667468510624 [ 7351.020839] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7351.042469] LustreError: 341614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7351.496359] Lustre: server umount lustre-MDT0001 complete [ 7358.948517] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:31:57 (1788881517) [ 7364.744261] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:32:03 (1788881523) [ 7372.514622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.017074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.156534] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7378.402133] LustreError: 346345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.421842] LustreError: 346345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7382.507780] LustreError: 346346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.644746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.909799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7388.502385] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7395.258502] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7400.586590] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7407.570936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7407.859229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7414.834239] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7422.372675] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7430.601694] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7432.102599] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7433.781756] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7435.509579] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7437.498272] Lustre: Failing over lustre-MDT0000 [ 7437.743768] Lustre: server umount lustre-MDT0000 complete [ 7438.824479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.845887] Lustre: Skipped 3 previous similar messages [ 7438.851376] LustreError: 346342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.866126] LustreError: 346342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7443.941941] LustreError: 347074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7443.962487] LustreError: 347074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7449.061151] LustreError: 346346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.079765] LustreError: 346346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7455.200601] Lustre: 339844:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788881599/real 1788881599] req@0000000083b2197c x1875778093517696/t0(0) o400->MGC192.168.204.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788881615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7455.216821] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.736901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.322677] LustreError: 347074:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7460.342317] LustreError: 347074:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7465.452732] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7a53726385ee5d48 [ 7465.467398] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo) [ 7465.752510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.801958] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7466.079918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7469.785115] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7471.112618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7471.143519] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7471.199893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7478.056474] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7479.248456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7482.359294] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7483.832717] Lustre: Failing over lustre-MDT0000 [ 7484.045370] Lustre: server umount lustre-MDT0000 complete [ 7486.442159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7486.442566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.444276] LustreError: 346346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.444286] LustreError: 346346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7486.488872] Lustre: Skipped 2 previous similar messages [ 7501.047634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.145864] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.323701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7501.371868] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7501.921793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7504.842928] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7506.415284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7506.420427] Lustre: Skipped 2 previous similar messages [ 7506.443483] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7506.469956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7512.626428] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7513.971882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7521.764790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.770556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.776268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7521.780312] Lustre: Skipped 2 previous similar messages [ 7521.803815] Lustre: Skipped 2 previous similar messages [ 7524.985644] Lustre: server umount lustre-MDT0000 complete [ 7526.882545] LustreError: 346340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.892212] LustreError: 346340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7528.518808] LustreError: 346327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881689 with bad export cookie 8814514667468513340 [ 7528.527571] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7528.533893] LustreError: 346327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7528.842275] Lustre: server umount lustre-MDT0001 complete [ 7542.936452] Lustre: server umount lustre-OST0000 complete [ 7553.438547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.290641] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7561.352329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7564.140861] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7568.706459] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7572.365247] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7577.052154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.293961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7577.297357] Lustre: Skipped 2 previous similar messages [ 7578.350738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7581.396457] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7585.809502] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7593.955048] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7593.959803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.967318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7599.741637] Lustre: server umount lustre-OST0000 complete [ 7602.148635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7602.152504] Lustre: Skipped 2 previous similar messages [ 7608.044156] Lustre: server umount lustre-MDT0000 complete [ 7610.071287] LustreError: 355462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881770 with bad export cookie 8814514667468514089 [ 7610.074253] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.083591] LustreError: 355462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7610.256333] Lustre: server umount lustre-MDT0001 complete [ 7615.923835] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7617.576679] Key type lgssc unregistered [ 7617.778554] LNet: 356211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7617.782906] LNetError: 356211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7617.790147] LNet: Removed LNI 192.168.204.133@tcp [ 7618.327615] Key type .llcrypt unregistered [ 7618.329549] Key type ._llcrypt unregistered [ 7633.777912] Key type ._llcrypt registered [ 7633.779518] Key type .llcrypt registered [ 7633.865279] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7634.416788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7634.476530] alg: No test for adler32 (adler32-zlib) [ 7635.449904] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 7635.651553] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7637.280220] Key type lgssc registered [ 7638.028071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7642.929551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7642.950347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7644.272193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.082868] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7652.470585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.677348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7655.021330] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7659.512411] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7663.007294] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7667.283684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.538893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7669.621697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7671.034814] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7675.086364] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.163722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.166089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.169564] Lustre: Skipped 2 previous similar messages [ 7688.171433] Lustre: Skipped 1 previous similar message [ 7690.988276] Lustre: server umount lustre-MDT0000 complete [ 7692.838699] LustreError: 360281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881853 with bad export cookie 16675392890946906462 [ 7692.841821] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.845205] LustreError: 360281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7693.025925] Lustre: server umount lustre-MDT0001 complete [ 7705.476714] Lustre: server umount lustre-OST0000 complete [ 7713.116947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.420831] LustreError: 361412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.441720] LustreError: 361412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7713.472697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.130543] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7718.881217] LustreError: 361413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.360701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.621103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7723.913411] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7728.447531] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7731.523990] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7735.542987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7735.745815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7738.805716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7739.254925] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7743.196648] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7752.162466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7752.163916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.165665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7752.165673] Lustre: Skipped 1 previous similar message [ 7752.175673] Lustre: Skipped 2 previous similar messages [ 7757.175835] Lustre: server umount lustre-MDT0000 complete [ 7757.281671] LustreError: 361413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.288918] LustreError: 361413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7758.885409] LustreError: 361393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881919 with bad export cookie 16675392890946907561 [ 7758.889082] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7758.891793] LustreError: 361393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7759.038778] Lustre: server umount lustre-MDT0001 complete [ 7770.981422] Lustre: server umount lustre-OST0000 complete [ 7777.156308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.364301] LustreError: 365319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7777.400034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7779.244944] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7782.755117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.600233] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7787.653153] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7789.876912] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7792.621876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7795.092406] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7798.156669] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7804.010094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7812.066691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.067725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7812.071657] Lustre: Skipped 1 previous similar message [ 7812.076182] Lustre: Skipped 3 previous similar messages [ 7816.557357] Lustre: server umount lustre-OST0000 complete [ 7818.721914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7818.725359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.738562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7824.547677] Lustre: server umount lustre-MDT0000 complete [ 7826.086665] LustreError: 365300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788881986 with bad export cookie 16675392890946908569 [ 7826.089876] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.092844] LustreError: 365300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7826.234194] Lustre: server umount lustre-MDT0001 complete [ 7831.029403] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7832.472955] Key type lgssc unregistered [ 7832.693901] LNet: 368867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7832.703074] LNetError: 368867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7832.716292] LNet: Removed LNI 192.168.204.133@tcp [ 7833.144311] Key type .llcrypt unregistered [ 7833.145862] Key type ._llcrypt unregistered [ 7844.776431] Key type ._llcrypt registered [ 7844.778130] Key type .llcrypt registered [ 7844.821472] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7845.203472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7845.223115] alg: No test for adler32 (adler32-zlib) [ 7846.100131] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 7846.200658] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7847.792225] Key type lgssc registered [ 7848.247071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7851.318377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7851.325576] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7851.329779] LustreError: 370282:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7851.340562] LustreError: 370282:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7851.344225] LustreError: 370282:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7851.347046] LustreError: 370282:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7851.350354] LustreError: 370282:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7854.393485] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.397347] LustreError: 370678:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7854.407291] LustreError: 370678:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7854.410805] LustreError: 370678:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7854.414211] LustreError: 370678:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7854.417567] LustreError: 370678:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7861.621705] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 7862.786337] Key type lgssc unregistered [ 7862.921499] LNet: 371679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7862.924861] LNetError: 371679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7862.931433] LNet: Removed LNI 192.168.204.133@tcp [ 7863.272587] Key type .llcrypt unregistered [ 7863.273676] Key type ._llcrypt unregistered [ 7869.875757] Key type ._llcrypt registered [ 7869.877208] Key type .llcrypt registered [ 7869.918563] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 7876.838103] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 7877.360140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.366489] alg: No test for adler32 (adler32-zlib) [ 7878.242300] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 7878.341503] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 7879.928150] Key type lgssc registered [ 7880.331990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7892.743869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7892.752344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.848325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7893.862311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7893.901924] Lustre: lustre-MDT0000: new disk, initializing [ 7893.936782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7893.945055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7895.538298] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7899.981968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7900.030067] Lustre: 374813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7900.059473] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7900.061801] Lustre: Skipped 1 previous similar message [ 7900.107146] Lustre: lustre-MDT0001: new disk, initializing [ 7900.150809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7900.165190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7900.169500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7901.680867] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7905.757199] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7908.005219] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7910.783279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7910.899403] Lustre: lustre-OST0000: new disk, initializing [ 7910.901923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7910.905221] Lustre: 376781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7910.940533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7913.205253] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7916.547413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7916.552753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7916.601266] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7917.450837] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7919.171525] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7919.265649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7920.934435] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7921.032368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7926.755411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7926.755892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7926.763892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7926.767269] Lustre: Skipped 1 previous similar message [ 7927.902280] Lustre: server umount lustre-OST0000 complete [ 7930.848884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7930.851588] LustreError: Skipped 1 previous similar message [ 7930.853323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.857142] Lustre: Skipped 1 previous similar message [ 7930.859427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.873402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7931.873754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.387264] Lustre: server umount lustre-MDT0000 complete [ 7936.670255] LustreError: 374806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882097 with bad export cookie 5480362552171921088 [ 7936.673550] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7936.675443] LustreError: 374806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7936.801905] Lustre: server umount lustre-MDT0001 complete [ 7939.704202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7940.276810] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:41:39 (1788882099) [ 7942.536972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.737750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7944.145502] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7947.004442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.115421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7948.468605] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7950.810491] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7952.581307] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7954.812242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.939746] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7959.266515] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7961.488327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.554539] Lustre: lustre-OST0001: new disk, initializing [ 7961.556436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7961.558643] Lustre: 382068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7961.588579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7961.591119] Lustre: Skipped 1 previous similar message [ 7963.143210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7963.146618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7963.185209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7963.548147] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 7967.192677] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7972.323444] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.326138] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7972.327933] Lustre: Skipped 1 previous similar message [ 7972.330067] Lustre: Skipped 1 previous similar message [ 7975.100796] Lustre: server umount lustre-OST0001 complete [ 7977.445735] LustreError: 381053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7977.450586] LustreError: 381053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7982.567340] LustreError: 381435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7982.574087] LustreError: 381435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7987.681418] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7987.682513] LustreError: 381053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7987.683561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.683760] LustreError: Skipped 1 previous similar message [ 7987.684645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7987.684650] Lustre: Skipped 1 previous similar message [ 7987.689582] LustreError: 381053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7992.804048] LustreError: 381056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.809778] LustreError: 381056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7993.367554] Lustre: server umount lustre-OST0000 complete [ 7997.922423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.923311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.927184] Lustre: Skipped 2 previous similar messages [ 7997.929878] Lustre: Skipped 4 previous similar messages [ 8000.869475] Lustre: server umount lustre-MDT0000 complete [ 8002.186460] LustreError: 383004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882163 with bad export cookie 5480362552171921970 [ 8002.189620] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.192212] LustreError: 383004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8008.383812] Lustre: server umount lustre-MDT0001 complete [ 8012.016135] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8013.116350] Key type lgssc unregistered [ 8013.250565] LNet: 384056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8013.253520] LNetError: 384056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8013.260346] LNet: Removed LNI 192.168.204.133@tcp [ 8013.596956] Key type .llcrypt unregistered [ 8013.598499] Key type ._llcrypt unregistered [ 8032.186377] Key type ._llcrypt registered [ 8032.187811] Key type .llcrypt registered [ 8032.226541] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8032.651438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8032.685922] alg: No test for adler32 (adler32-zlib) [ 8033.551458] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8033.649605] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8035.248153] Key type lgssc registered [ 8035.652325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8038.460912] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8038.468749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.565419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8039.573433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8039.637315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8041.099298] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8044.081502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8044.114649] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8044.125036] Lustre: 386828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8044.228791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8045.629864] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8048.088878] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8050.000776] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8051.681530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.682212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.686209] Lustre: Skipped 1 previous similar message [ 8051.689232] Lustre: Skipped 1 previous similar message [ 8056.801562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.803711] Lustre: Skipped 1 previous similar message [ 8056.854908] Lustre: server umount lustre-MDT0000 complete [ 8058.177430] LustreError: 386820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882219 with bad export cookie 1371254714601111133 [ 8058.180675] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8058.181102] LustreError: 386820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8058.287568] Lustre: server umount lustre-MDT0001 complete [ 8061.594032] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:43:40 (1788882220) [ 8063.879253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8064.055227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.483428] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8068.367508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.484838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8069.873197] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8072.331942] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8074.222217] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8076.537911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8076.616762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8076.680338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8078.692242] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8081.116113] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8084.824645] Lustre: Permanently deactivating lustre-OST0000 [ 8084.830178] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8086.323433] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8086.765400] LustreError: 392174:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8086.770145] LustreError: 392174:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8086.773722] LustreError: 392174:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8087.414479] LustreError: 392222:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8087.417842] LustreError: 392222:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8094.155761] Lustre: server umount lustre-OST0000 complete [ 8095.043981] LustreError: 392471:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 8095.047813] LustreError: 392471:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8099.296917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.300088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.305083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.929061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8104.931906] Lustre: Skipped 2 previous similar messages [ 8105.976851] Lustre: server umount lustre-MDT0000 complete [ 8107.159040] LustreError: 389361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882268 with bad export cookie 1371254714601111700 [ 8107.161629] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8107.162949] LustreError: 389361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8117.777073] Lustre: server umount lustre-MDT0001 complete [ 8120.584181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8121.931132] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8122.831204] LustreError: 393630:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8124.950745] Lustre: 393968:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.133@tcp [ 8125.609365] Lustre: server umount MGS complete [ 8129.478386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8129.656521] LustreError: 394692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8129.662319] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8129.664239] LustreError: 394692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8129.666511] Lustre: Skipped 1 previous similar message [ 8129.694741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8131.050428] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8133.917786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8134.016857] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8135.402716] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8137.869354] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.632752] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8140.262888] Lustre: Permanently reactivating lustre-OST0000 [ 8140.267126] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8140.268635] Lustre: Skipped 2 previous similar messages [ 8166.880146] LustreError: 396519:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8170.017405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8170.125292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.127718] Lustre: Skipped 1 previous similar message [ 8172.088127] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8174.442976] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8186.850785] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8186.851365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8186.853387] LustreError: Skipped 1 previous similar message [ 8186.853945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8186.857141] Lustre: Skipped 2 previous similar messages [ 8188.238697] Lustre: server umount lustre-OST0000 complete [ 8190.432929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8190.436548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.567339] Lustre: server umount lustre-MDT0000 complete [ 8191.968848] LustreError: 394693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8191.973905] LustreError: 394693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8192.754242] LustreError: 396263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882353 with bad export cookie 1371254714601112722 [ 8192.756532] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8192.758282] LustreError: 396263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8192.859606] Lustre: server umount lustre-MDT0001 complete [ 8196.360377] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8197.372891] Key type lgssc unregistered [ 8197.501504] LNet: 399409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8197.504892] LNetError: 399409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8197.514324] LNet: Removed LNI 192.168.204.133@tcp [ 8197.817063] Key type .llcrypt unregistered [ 8197.818547] Key type ._llcrypt unregistered [ 8203.663087] Key type ._llcrypt registered [ 8203.663910] Key type .llcrypt registered [ 8203.699312] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 8209.943328] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8210.219667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8210.228288] alg: No test for adler32 (adler32-zlib) [ 8211.079113] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8211.165765] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8212.744128] Key type lgssc registered [ 8213.077447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8223.883903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8223.889641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.968812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8224.979901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8225.013281] Lustre: lustre-MDT0000: new disk, initializing [ 8225.034714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.040368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8226.296711] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8230.232408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.271905] Lustre: 402536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8230.292612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8230.294286] Lustre: Skipped 1 previous similar message [ 8230.325499] Lustre: lustre-MDT0001: new disk, initializing [ 8230.353149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.364207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8230.367547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8231.604477] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8235.119285] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8236.889670] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8239.071588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8239.185449] Lustre: lustre-OST0000: new disk, initializing [ 8239.187337] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8239.190822] Lustre: 404501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8239.224929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.159372] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8244.220306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8244.223691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8244.259514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8244.936439] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8246.378180] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8246.470753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8247.777032] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8247.841927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8249.313917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8249.314838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.316551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8249.317747] LustreError: Skipped 1 previous similar message [ 8249.321566] Lustre: Skipped 1 previous similar message [ 8250.519418] Lustre: server umount lustre-OST0000 complete [ 8254.433543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.434321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.438907] Lustre: Skipped 1 previous similar message [ 8254.441130] Lustre: Skipped 1 previous similar message [ 8257.823512] Lustre: server umount lustre-MDT0000 complete [ 8259.066261] LustreError: 402526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882419 with bad export cookie 674266974458935816 [ 8259.068888] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8259.070311] LustreError: 402526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8259.183146] Lustre: server umount lustre-MDT0001 complete [ 8261.875920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8262.424335] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:47:01 (1788882421) [ 8264.643197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.809551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8266.095224] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8268.895627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.009471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8270.334124] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8272.665362] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8274.467263] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8276.583849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8278.539682] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8280.859092] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.481627] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8292.520052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8294.368771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8294.371552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8294.375139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8294.377050] Lustre: Skipped 1 previous similar message [ 8296.418952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8296.419464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.427568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8299.975343] Lustre: server umount lustre-OST0000 complete [ 8301.537152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8301.537892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8301.544293] Lustre: Skipped 1 previous similar message [ 8301.545800] Lustre: Skipped 1 previous similar message [ 8307.281474] Lustre: server umount lustre-MDT0000 complete [ 8308.410169] LustreError: 406936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882469 with bad export cookie 674266974458936677 [ 8308.412698] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8308.413447] LustreError: 406936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8308.522060] Lustre: server umount lustre-MDT0001 complete [ 8312.006920] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8312.965430] Key type lgssc unregistered [ 8313.100347] LNet: 410697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8313.102634] LNetError: 410697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8313.111292] LNet: Removed LNI 192.168.204.133@tcp [ 8313.425215] Key type .llcrypt unregistered [ 8313.426401] Key type ._llcrypt unregistered [ 8320.987701] Key type ._llcrypt registered [ 8320.988942] Key type .llcrypt registered [ 8321.023746] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8321.127195] Key type .llcrypt unregistered [ 8321.128377] Key type ._llcrypt unregistered [ 8327.725612] Key type ._llcrypt registered [ 8327.726758] Key type .llcrypt registered [ 8327.759146] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8328.053921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8328.067941] alg: No test for adler32 (adler32-zlib) [ 8328.914089] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8328.988431] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8330.568152] Key type lgssc registered [ 8330.888499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8333.209826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8333.214512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.336624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.526478] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8338.091834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.191951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8339.918229] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8342.114356] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8343.521205] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8343.525857] LustreError: 412878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8344.257269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.367365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8345.385306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8345.616691] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8349.664910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8349.666890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.670775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.690663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8350.690893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.693043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.513378] Lustre: server umount lustre-OST0000 complete [ 8355.809473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8355.810184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8355.813043] Lustre: Skipped 1 previous similar message [ 8355.815523] Lustre: Skipped 1 previous similar message [ 8360.929522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8360.931506] Lustre: Skipped 2 previous similar messages [ 8361.942238] Lustre: server umount lustre-MDT0000 complete [ 8363.240257] LustreError: 412858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882524 with bad export cookie 2321435019785905295 [ 8363.244284] LustreError: 412858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8363.245021] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8363.375678] Lustre: server umount lustre-MDT0001 complete [ 8366.834861] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8367.784715] Key type lgssc unregistered [ 8367.900369] LNet: 416023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8367.903773] LNetError: 416023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8367.911295] LNet: Removed LNI 192.168.204.133@tcp [ 8368.230853] Key type .llcrypt unregistered [ 8368.231866] Key type ._llcrypt unregistered [ 8377.396916] Key type ._llcrypt registered [ 8377.398837] Key type .llcrypt registered [ 8377.446490] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8377.804415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8377.812831] alg: No test for adler32 (adler32-zlib) [ 8378.676484] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8378.770282] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8380.352166] Key type lgssc registered [ 8380.706938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8383.252082] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8383.259740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8398.688313] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8403.808278] LustreError: 417386:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.133@tcp: failed processing log, type 4: rc = -110 [ 8430.432210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.353868] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8434.877849] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8437.159852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.323406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.351113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8438.617364] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8441.404769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.534070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8442.871956] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8445.369111] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8447.153886] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8451.044173] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8451.045408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8451.049347] Lustre: Skipped 1 previous similar message [ 8451.054243] Lustre: Skipped 1 previous similar message [ 8455.565953] Lustre: server umount lustre-OST0000 complete [ 8457.184895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8457.187214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.190959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8461.281211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8461.281511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.934696] Lustre: server umount lustre-MDT0000 complete [ 8464.150739] LustreError: 417392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882625 with bad export cookie 14766226612752602276 [ 8464.153977] LustreError: 417392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8464.154858] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8464.270773] Lustre: server umount lustre-MDT0001 complete [ 8468.094487] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8469.160623] Key type lgssc unregistered [ 8469.298353] LNet: 420952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8469.300650] LNetError: 420952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8469.306307] LNet: Removed LNI 192.168.204.133@tcp [ 8469.615626] Key type .llcrypt unregistered [ 8469.616885] Key type ._llcrypt unregistered [ 8478.916511] Key type ._llcrypt registered [ 8478.917516] Key type .llcrypt registered [ 8478.955900] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8479.355151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8479.403075] alg: No test for adler32 (adler32-zlib) [ 8480.252232] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8480.344840] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8481.928195] Key type lgssc registered [ 8482.286317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8484.880172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8484.886080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.038086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.475073] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8490.344391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8490.472524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.801210] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8494.201466] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8496.002602] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8498.240501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8498.357574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8500.299025] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8502.801689] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8510.631815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8516.066828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.067220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8516.071762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8516.076661] Lustre: Skipped 2 previous similar messages [ 8518.243292] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.33@tcp (stopping) [ 8520.154742] Lustre: server umount lustre-MDT0000 complete [ 8520.896938] LustreError: 423078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.33@tcp arrived at 1788882681 with bad export cookie 3518044652462226455 [ 8521.186442] LustreError: 422362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8521.193921] LustreError: 422362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8521.536332] LustreError: 422323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882682 with bad export cookie 3518044652462225832 [ 8521.540557] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8521.541460] LustreError: 422323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8526.305439] LustreError: 422343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.305845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8526.305995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8526.306017] Lustre: Skipped 2 previous similar messages [ 8526.306643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8526.310981] LustreError: 422343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8527.627335] Lustre: server umount lustre-OST0000 complete [ 8529.816551] Lustre: server umount lustre-MDT0001 complete [ 8533.593499] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8534.546886] Key type lgssc unregistered [ 8534.674421] LNet: 426044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8534.676951] LNetError: 426044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8534.685314] LNet: Removed LNI 192.168.204.133@tcp [ 8534.987630] Key type .llcrypt unregistered [ 8534.988749] Key type ._llcrypt unregistered [ 8543.997844] Key type ._llcrypt registered [ 8543.998898] Key type .llcrypt registered [ 8544.043261] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8544.608252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8544.621915] alg: No test for adler32 (adler32-zlib) [ 8545.473165] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8545.561809] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8547.144107] Key type lgssc registered [ 8547.502054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8549.999895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8550.007530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.152605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8552.529924] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8555.270365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.391403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8556.804466] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8559.107568] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8560.917799] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8563.131527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8563.238755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8565.108151] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8567.495764] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8575.523613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8580.581560] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.585636] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8580.587249] Lustre: Skipped 1 previous similar message [ 8583.266436] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 8585.378098] Lustre: server umount lustre-MDT0001 complete [ 8585.697651] LustreError: 428172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.702808] LustreError: 428172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8590.817794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8590.817821] LustreError: 427434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.818899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.818904] Lustre: Skipped 1 previous similar message [ 8590.819416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8593.101304] Lustre: server umount lustre-OST0000 complete [ 8594.403418] Lustre: server umount lustre-MDT0000 complete [ 8598.966676] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8599.944460] Key type lgssc unregistered [ 8600.066377] LNet: 431137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8600.068705] LNetError: 431137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8600.076252] LNet: Removed LNI 192.168.204.133@tcp [ 8600.367934] Key type .llcrypt unregistered [ 8600.368966] Key type ._llcrypt unregistered [ 8608.413715] Key type ._llcrypt registered [ 8608.414767] Key type .llcrypt registered [ 8608.452213] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 8614.436127] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8614.805908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8614.828382] alg: No test for adler32 (adler32-zlib) [ 8615.677225] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8615.762022] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8617.344199] Key type lgssc registered [ 8617.725176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8629.881758] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8635.152370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.158257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8636.241525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8636.251630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8636.288436] Lustre: lustre-MDT0000: new disk, initializing [ 8636.308040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.313696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8637.720177] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8643.711120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8643.817454] Lustre: lustre-OST0000: new disk, initializing [ 8643.819341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8643.820976] Lustre: Skipped 1 previous similar message [ 8643.822879] Lustre: 436139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8643.848252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8644.623237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8644.628120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8644.639754] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8645.887373] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8649.731694] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8655.144600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8655.185445] Lustre: 435119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8655.215657] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8655.253597] Lustre: lustre-MDT0001: new disk, initializing [ 8655.299224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.314108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8655.317576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8656.778878] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8660.872724] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8660.951943] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8662.379206] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8663.510897] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 8667.623381] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8667.623441] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8667.630208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8667.634703] Lustre: Skipped 1 previous similar message [ 8668.779355] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.33@tcp (stopping) [ 8671.319664] Lustre: server umount lustre-MDT0001 complete [ 8677.860225] LustreError: 435127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.868694] LustreError: 435127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8682.981381] LustreError: 435128:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8682.995226] LustreError: 435128:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8685.537042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8685.540678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8685.546303] Lustre: Skipped 1 previous similar message [ 8685.548689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.097522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8688.098104] LustreError: 435132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8693.217464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.218439] LustreError: 435131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.336209] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8698.338495] LustreError: 435132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8698.403175] Lustre: server umount lustre-OST0000 complete [ 8714.208129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8714.305861] Lustre: server umount lustre-MDT0000 complete [ 8718.899057] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8719.907990] Key type lgssc unregistered [ 8720.048299] LNet: 439891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8720.050569] LNetError: 439891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8720.059273] LNet: Removed LNI 192.168.204.133@tcp [ 8720.372120] Key type .llcrypt unregistered [ 8720.373619] Key type ._llcrypt unregistered [ 8733.337727] Key type ._llcrypt registered [ 8733.338984] Key type .llcrypt registered [ 8733.377530] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 8733.485251] Key type .llcrypt unregistered [ 8733.486817] Key type ._llcrypt unregistered [ 8740.100458] Key type ._llcrypt registered [ 8740.101815] Key type .llcrypt registered [ 8740.140903] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 8740.613572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8740.628840] alg: No test for adler32 (adler32-zlib) [ 8741.489318] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 8741.586059] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 8743.168229] Key type lgssc registered [ 8743.569145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8752.312511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8752.318027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.424794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8753.440650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8753.483415] Lustre: lustre-MDT0000: new disk, initializing [ 8753.506598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8753.512597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8754.913361] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8759.058359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.102379] Lustre: 443233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8759.126867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8759.129020] Lustre: Skipped 1 previous similar message [ 8759.173354] Lustre: lustre-MDT0001: new disk, initializing [ 8759.204746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.217506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8759.222265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8760.722446] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8764.435634] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8766.457117] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8768.748875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8768.878536] Lustre: lustre-OST0000: new disk, initializing [ 8768.881673] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8768.884432] Lustre: 445201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8768.917348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8770.880741] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8774.606538] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8776.186357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8776.189857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8776.221502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8784.865123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8784.867384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.871570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.402608] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.403233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.407050] Lustre: Skipped 1 previous similar message [ 8786.408920] Lustre: Skipped 1 previous similar message [ 8788.118860] Lustre: server umount lustre-MDT0000 complete [ 8789.369154] LustreError: 443227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788882950 with bad export cookie 4355985762947683456 [ 8789.372402] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8789.372542] LustreError: 443227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8789.476721] Lustre: server umount lustre-MDT0001 complete [ 8800.925904] Lustre: server umount lustre-OST0000 complete [ 8804.379114] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:56:03 (1788882963) [ 8806.870203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.038135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8808.327937] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8811.068513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.186725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8812.533704] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8814.854346] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8816.706378] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8831.974882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8832.092413] Lustre: Found index 0 for lustre-OST0000, updating log [ 8832.172955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.403346] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 8836.988688] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8852.650853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8862.177155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8862.179699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.184350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8862.187116] Lustre: Skipped 1 previous similar message [ 8865.108703] Lustre: server umount lustre-OST0000 complete [ 8878.049142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8878.051076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8878.055235] Lustre: Skipped 1 previous similar message [ 8878.057298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8878.059479] Lustre: Skipped 3 previous similar messages [ 8884.705510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.708494] Lustre: Skipped 3 previous similar messages [ 8890.080294] LNet: 1 local NIs in recovery (showing 1): 192.168.204.133@tcp [ 8891.872161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8891.928951] Lustre: server umount lustre-MDT0000 complete [ 8893.163794] LustreError: 447212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883054 with bad export cookie 4355985762947684828 [ 8893.167337] LustreError: 447212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8893.168472] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.299612] Lustre: server umount lustre-MDT0001 complete [ 8896.109163] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:57:35 (1788883055) [ 9478.564290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9478.770147] LustreError: 453665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.778914] LustreError: 453665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9478.803846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9480.389555] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9483.720881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9483.868845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9485.508543] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9488.455722] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9490.618776] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9506.084926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9506.202090] Lustre: Found index 0 for lustre-OST0000, updating log [ 9506.275526] LustreError: 455526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.354929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9508.703967] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9510.442096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9511.580388] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9515.491896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9515.492473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9515.498957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9515.500846] Lustre: Skipped 2 previous similar messages [ 9515.506288] Lustre: Skipped 5 previous similar messages [ 9518.428395] Lustre: server umount lustre-OST0000 complete [ 9520.866330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.867927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9520.871037] Lustre: Skipped 1 previous similar message [ 9520.876197] Lustre: Skipped 1 previous similar message [ 9522.913579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.916291] Lustre: Skipped 2 previous similar messages [ 9525.981129] Lustre: server umount lustre-MDT0000 complete [ 9525.985680] LustreError: 454394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9525.992806] LustreError: 454394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9527.528180] LustreError: 453647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883688 with bad export cookie 4355985762947685899 [ 9527.532294] LustreError: 453647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9527.533145] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9527.673456] Lustre: server umount lustre-MDT0001 complete [ 9530.080357] LNet: 1 local NIs in recovery (showing 1): 192.168.204.133@tcp [ 9545.944115] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9550.188947] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9565.905519] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:08:45 (1788883725) [ 9573.289974] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9574.536136] Key type lgssc unregistered [ 9574.687453] LNet: 460297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9574.690929] LNetError: 460297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9635.110291] LNet: Removed LNI 192.168.204.133@tcp [ 9635.434819] Key type .llcrypt unregistered [ 9635.436549] Key type ._llcrypt unregistered [ 9642.456816] Key type ._llcrypt registered [ 9642.458086] Key type .llcrypt registered [ 9642.502183] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9642.847576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9642.876092] alg: No test for adler32 (adler32-zlib) [ 9643.744346] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 9643.849841] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 9645.440177] Key type lgssc registered [ 9645.930940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9647.802540] LNet: Added LNI 192.168.204.133@tcp100 [8/256/0/180] [ 9648.175833] LNet: Added LNI 192.168.204.133@tcp101 [8/256/0/180] [ 9648.543473] LNet: Added LNI 192.168.204.133@tcp102 [8/256/0/180] [ 9648.916990] LNet: Added LNI 192.168.204.133@tcp103 [8/256/0/180] [ 9649.281325] LNet: Added LNI 192.168.204.133@tcp104 [8/256/0/180] [ 9649.642792] LNet: Added LNI 192.168.204.133@tcp105 [8/256/0/180] [ 9650.030782] LNet: Added LNI 192.168.204.133@tcp106 [8/256/0/180] [ 9650.441976] LNet: Added LNI 192.168.204.133@tcp107 [8/256/0/180] [ 9650.831475] LNet: Added LNI 192.168.204.133@tcp108 [8/256/0/180] [ 9651.201677] LNet: Added LNI 192.168.204.133@tcp109 [8/256/0/180] [ 9651.582946] LNet: Added LNI 192.168.204.133@tcp110 [8/256/0/180] [ 9651.970337] LNet: Added LNI 192.168.204.133@tcp111 [8/256/0/180] [ 9652.367587] LNet: Added LNI 192.168.204.133@tcp112 [8/256/0/180] [ 9652.724713] LNet: Added LNI 192.168.204.133@tcp113 [8/256/0/180] [ 9653.079089] LNet: Added LNI 192.168.204.133@tcp114 [8/256/0/180] [ 9653.454543] LNet: Added LNI 192.168.204.133@tcp115 [8/256/0/180] [ 9653.838505] LNet: Added LNI 192.168.204.133@tcp116 [8/256/0/180] [ 9654.191722] LNet: Added LNI 192.168.204.133@tcp117 [8/256/0/180] [ 9654.574761] LNet: Added LNI 192.168.204.133@tcp118 [8/256/0/180] [ 9654.930190] LNet: Added LNI 192.168.204.133@tcp119 [8/256/0/180] [ 9655.263056] LNet: Added LNI 192.168.204.133@tcp120 [8/256/0/180] [ 9655.628377] LNet: Added LNI 192.168.204.133@tcp121 [8/256/0/180] [ 9656.026617] LNet: Added LNI 192.168.204.133@tcp122 [8/256/0/180] [ 9656.418079] LNet: Added LNI 192.168.204.133@tcp123 [8/256/0/180] [ 9656.794243] LNet: Added LNI 192.168.204.133@tcp124 [8/256/0/180] [ 9657.177668] LNet: Added LNI 192.168.204.133@tcp125 [8/256/0/180] [ 9657.591425] LNet: Added LNI 192.168.204.133@tcp126 [8/256/0/180] [ 9658.018600] LNet: Added LNI 192.168.204.133@tcp127 [8/256/0/180] [ 9658.449923] LNet: Added LNI 192.168.204.133@tcp128 [8/256/0/180] [ 9658.840243] LNet: Added LNI 192.168.204.133@tcp129 [8/256/0/180] [ 9659.258103] LNet: Added LNI 192.168.204.133@tcp130 [8/256/0/180] [ 9659.637768] LNet: Added LNI 192.168.204.133@tcp131 [8/256/0/180] [ 9660.029802] LNet: Added LNI 192.168.204.133@tcp132 [8/256/0/180] [ 9660.387285] LNet: Added LNI 192.168.204.133@tcp133 [8/256/0/180] [ 9660.752247] LNet: Added LNI 192.168.204.133@tcp134 [8/256/0/180] [ 9661.095865] LNet: Added LNI 192.168.204.133@tcp135 [8/256/0/180] [ 9663.727192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9663.733364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9664.772818] Lustre: 463291:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [ 9664.799242] LNetError: 460752:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9664.805940] LNetError: Protocol error connecting to 192.168.204.133@tcp122 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9664.811357] LNetError: Skipped 7 previous similar messages [ 9664.864968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9664.878687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9664.919869] Lustre: lustre-MDT0000: new disk, initializing [ 9664.946456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9664.953589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9665.376348] LNet: 9 local NIs in recovery (showing 9): 192.168.204.133@tcp126, 192.168.204.133@tcp127, 192.168.204.133@tcp128, 192.168.204.133@tcp129, 192.168.204.133@tcp130, 192.168.204.133@tcp122, 192.168.204.133@tcp131, 192.168.204.133@tcp132, 192.168.204.133@tcp133 [ 9665.377994] LNetError: 463317:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9665.391138] LNetError: 463317:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [ 9665.394678] LNetError: Protocol error connecting to 192.168.204.133@tcp103 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9665.398819] LNetError: Skipped 5 previous similar messages [ 9666.517638] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9667.426697] LNetError: 460753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9667.430640] LNetError: 460753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9667.434198] LNetError: Protocol error connecting to 192.168.204.133@tcp131 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9667.437698] LNetError: Skipped 4 previous similar messages [ 9670.852483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9670.908331] Lustre: 463307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9670.933316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9670.935954] Lustre: Skipped 1 previous similar message [ 9670.971679] Lustre: lustre-MDT0001: new disk, initializing [ 9670.980627] LNetError: 460754:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9670.984072] LNetError: Protocol error connecting to 192.168.204.133@tcp134 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9670.984992] Lustre: 464114:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [ 9670.990550] Lustre: 464114:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9671.039166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9671.051841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9671.055441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9672.183462] Lustre: 464308:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [ 9672.187558] Lustre: 464308:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 9672.541457] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9676.485110] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9677.665895] LNetError: 463418:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9677.670181] LNetError: 463418:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9677.674146] LNetError: Protocol error connecting to 192.168.204.133@tcp134 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9677.681622] LNetError: Skipped 6 previous similar messages [ 9678.762619] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9681.552478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9681.677127] Lustre: lustre-OST0000: new disk, initializing [ 9681.679763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9681.684206] Lustre: 465284:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9681.729350] Lustre: 465289:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.133@tcp: can't add NID 192.168.204.133@tcp130: rc = -75 [ 9681.732900] Lustre: 465289:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 9681.755855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9682.148450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9682.152365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9682.205377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9683.960038] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9685.857315] LNetError: 463317:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9685.861709] LNetError: Protocol error connecting to 192.168.204.133@tcp134 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9688.195243] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9692.641760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9692.642599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.644813] LustreError: Skipped 1 previous similar message [ 9692.651289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9692.654305] Lustre: Skipped 1 previous similar message [ 9696.855996] Lustre: server umount lustre-OST0000 complete [ 9701.857077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9701.860386] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9701.865780] Lustre: Skipped 1 previous similar message [ 9701.868545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.881454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9702.881674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.266072] LNetError: 460752:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.133 [ 9703.270475] LNetError: 460752:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 9703.274484] LNetError: Protocol error connecting to 192.168.204.133@tcp123 at host 192.168.204.133:988: is it running a compatible version of Lustre? [ 9703.278829] LNetError: Skipped 7 previous similar messages [ 9704.353702] Lustre: server umount lustre-MDT0000 complete [ 9705.728231] LustreError: 463299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788883866 with bad export cookie 10502205480126519606 [ 9705.733054] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.733119] LustreError: 463299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9705.850934] Lustre: server umount lustre-MDT0001 complete [ 9709.716239] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9710.811351] Key type lgssc unregistered [ 9710.986727] LNet: 467195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9710.990608] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9710.993691] LNet: Removed LNI 192.168.204.133@tcp [ 9712.032158] LNet: Removed LNI 192.168.204.133@tcp100 [ 9712.034821] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9712.038319] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9712.041969] LNet: Removed LNI 192.168.204.133@tcp101 [ 9713.056097] LNet: Removed LNI 192.168.204.133@tcp102 [ 9713.057294] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9713.059795] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9713.061989] LNet: Removed LNI 192.168.204.133@tcp103 [ 9714.080155] LNet: Removed LNI 192.168.204.133@tcp104 [ 9715.104095] LNet: Removed LNI 192.168.204.133@tcp105 [ 9715.106033] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9715.108604] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9715.111827] LNet: Removed LNI 192.168.204.133@tcp106 [ 9716.128175] LNet: Removed LNI 192.168.204.133@tcp107 [ 9717.152145] LNet: Removed LNI 192.168.204.133@tcp108 [ 9718.176335] LNet: Removed LNI 192.168.204.133@tcp109 [ 9719.200110] LNet: Removed LNI 192.168.204.133@tcp110 [ 9719.202077] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9719.205039] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9719.207691] LNet: Removed LNI 192.168.204.133@tcp111 [ 9720.224914] LNet: Removed LNI 192.168.204.133@tcp112 [ 9721.248653] LNet: Removed LNI 192.168.204.133@tcp113 [ 9722.272298] LNet: Removed LNI 192.168.204.133@tcp114 [ 9723.296178] LNet: Removed LNI 192.168.204.133@tcp115 [ 9724.320187] LNet: Removed LNI 192.168.204.133@tcp116 [ 9725.344174] LNet: Removed LNI 192.168.204.133@tcp117 [ 9726.368143] LNet: Removed LNI 192.168.204.133@tcp118 [ 9727.392204] LNet: Removed LNI 192.168.204.133@tcp119 [ 9727.397034] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9727.402016] LNetError: 467195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9727.405360] LNet: Removed LNI 192.168.204.133@tcp120 [ 9728.416171] LNet: Removed LNI 192.168.204.133@tcp121 [ 9729.440166] LNet: Removed LNI 192.168.204.133@tcp122 [ 9730.464132] LNet: Removed LNI 192.168.204.133@tcp123 [ 9731.488160] LNet: Removed LNI 192.168.204.133@tcp124 [ 9732.512170] LNet: Removed LNI 192.168.204.133@tcp125 [ 9733.536152] LNet: Removed LNI 192.168.204.133@tcp126 [ 9734.560146] LNet: Removed LNI 192.168.204.133@tcp127 [ 9735.584197] LNet: Removed LNI 192.168.204.133@tcp128 [ 9736.608155] LNet: Removed LNI 192.168.204.133@tcp129 [ 9737.632159] LNet: Removed LNI 192.168.204.133@tcp130 [ 9738.656135] LNet: Removed LNI 192.168.204.133@tcp131 [ 9739.680140] LNet: Removed LNI 192.168.204.133@tcp132 [ 9740.704146] LNet: Removed LNI 192.168.204.133@tcp133 [ 9741.728145] LNet: Removed LNI 192.168.204.133@tcp134 [ 9742.757863] LNet: Removed LNI 192.168.204.133@tcp135 [ 9743.052583] Key type .llcrypt unregistered [ 9743.054285] Key type ._llcrypt unregistered [ 9751.365338] Key type ._llcrypt registered [ 9751.366597] Key type .llcrypt registered [ 9751.415375] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [ 9758.256636] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9758.724714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9758.736031] alg: No test for adler32 (adler32-zlib) [ 9759.612920] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 9759.726434] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 9761.328174] Key type lgssc registered [ 9761.722325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9774.759407] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:12:14 (1788883934) [ 9781.685162] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [ 9783.017967] Key type lgssc unregistered [ 9783.167653] LNet: 471253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9783.172088] LNetError: 471253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9783.180475] LNet: Removed LNI 192.168.204.133@tcp [ 9783.564283] Key type .llcrypt unregistered [ 9783.566177] Key type ._llcrypt unregistered [ 9791.272653] Key type ._llcrypt registered [ 9791.273775] Key type .llcrypt registered [ 9791.318193] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9791.779211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9791.809865] alg: No test for adler32 (adler32-zlib) [ 9792.691648] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 9792.798550] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [ 9794.400327] Key type lgssc registered [ 9794.945702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9798.833372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9798.840822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.946212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9799.961883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9800.001424] Lustre: lustre-MDT0000: new disk, initializing [ 9800.040978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9800.050463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9801.534492] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9806.312770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9806.360140] Lustre: 472440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9806.395173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9806.397314] Lustre: Skipped 1 previous similar message [ 9806.434266] Lustre: lustre-MDT0001: new disk, initializing [ 9806.473180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9806.487795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9806.492953] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9808.119267] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9812.269944] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9814.400755] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9817.094956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9817.221967] Lustre: lustre-OST0000: new disk, initializing [ 9817.224928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9817.228566] Lustre: 474406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9817.267547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9818.953747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9818.958209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9819.005324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9819.623928] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9823.644822] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9824.996332] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing lnet_if_list [ 9826.956509] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9826.971838] LNet: Added LNI 192.168.204.133@tcp10 [8/256/0/180] [ 9829.034613] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9829.048981] LNet: Added LNI 192.168.204.133@tcp11 [8/256/0/180] [ 9830.668582] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9830.687881] LNet: Added LNI 192.168.204.133@tcp12 [8/256/0/180] [ 9832.454049] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9832.472579] LNet: Added LNI 192.168.204.133@tcp13 [8/256/0/180] [ 9834.321179] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9834.335422] LNet: Added LNI 192.168.204.133@tcp14 [8/256/0/180] [ 9836.099750] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9836.113505] LNet: Added LNI 192.168.204.133@tcp15 [8/256/0/180] [ 9837.801752] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9837.814754] LNet: Added LNI 192.168.204.133@tcp16 [8/256/0/180] [ 9839.692418] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9839.706307] LNet: Added LNI 192.168.204.133@tcp17 [8/256/0/180] [ 9841.734112] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9841.748562] LNet: Added LNI 192.168.204.133@tcp18 [8/256/0/180] [ 9843.661994] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9843.674550] LNet: Added LNI 192.168.204.133@tcp19 [8/256/0/180] [ 9845.590522] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9845.603402] LNet: Added LNI 192.168.204.133@tcp20 [8/256/0/180] [ 9847.652267] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9847.666301] LNet: Added LNI 192.168.204.133@tcp21 [8/256/0/180] [ 9849.792160] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9849.806902] LNet: Added LNI 192.168.204.133@tcp22 [8/256/0/180] [ 9852.246429] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9852.274023] LNet: Added LNI 192.168.204.133@tcp23 [8/256/0/180] [ 9854.670590] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9854.686698] LNet: Added LNI 192.168.204.133@tcp24 [8/256/0/180] [ 9857.070171] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9857.085045] LNet: Added LNI 192.168.204.133@tcp25 [8/256/0/180] [ 9859.342278] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9859.355359] LNet: Added LNI 192.168.204.133@tcp26 [8/256/0/180] [ 9861.590548] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9861.604553] LNet: Added LNI 192.168.204.133@tcp27 [8/256/0/180] [ 9863.895426] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9863.915609] LNet: Added LNI 192.168.204.133@tcp28 [8/256/0/180] [ 9866.265062] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9866.277854] LNet: Added LNI 192.168.204.133@tcp29 [8/256/0/180] [ 9868.583458] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9868.596928] LNet: Added LNI 192.168.204.133@tcp30 [8/256/0/180] [ 9870.835951] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9870.849971] LNet: Added LNI 192.168.204.133@tcp31 [8/256/0/180] [ 9873.012764] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9873.024440] LNet: Added LNI 192.168.204.133@tcp32 [8/256/0/180] [ 9875.258032] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9875.270923] LNet: Added LNI 192.168.204.133@tcp33 [8/256/0/180] [ 9877.648922] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9877.662982] LNet: Added LNI 192.168.204.133@tcp34 [8/256/0/180] [ 9879.986936] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9880.000617] LNet: Added LNI 192.168.204.133@tcp35 [8/256/0/180] [ 9882.611516] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9882.625496] LNet: Added LNI 192.168.204.133@tcp36 [8/256/0/180] [ 9885.439980] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9885.452560] LNet: Added LNI 192.168.204.133@tcp37 [8/256/0/180] [ 9888.416680] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9888.432038] LNet: Added LNI 192.168.204.133@tcp38 [8/256/0/180] [ 9891.310557] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9891.324292] LNet: Added LNI 192.168.204.133@tcp39 [8/256/0/180] [ 9894.159801] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9894.176124] LNet: Added LNI 192.168.204.133@tcp40 [8/256/0/180] [ 9897.036513] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9897.051963] LNet: Added LNI 192.168.204.133@tcp41 [8/256/0/180] [ 9899.802416] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9899.817383] LNet: Added LNI 192.168.204.133@tcp42 [8/256/0/180] [ 9902.576285] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9902.589267] LNet: Added LNI 192.168.204.133@tcp43 [8/256/0/180] [ 9905.273674] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9905.287840] LNet: Added LNI 192.168.204.133@tcp44 [8/256/0/180] [ 9908.214935] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9908.228029] LNet: Added LNI 192.168.204.133@tcp45 [8/256/0/180] [ 9911.195054] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9911.209076] LNet: Added LNI 192.168.204.133@tcp46 [8/256/0/180] [ 9914.330526] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9914.345348] LNet: Added LNI 192.168.204.133@tcp47 [8/256/0/180] [ 9917.396717] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9917.411114] LNet: Added LNI 192.168.204.133@tcp48 [8/256/0/180] [ 9920.428240] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9920.442980] LNet: Added LNI 192.168.204.133@tcp49 [8/256/0/180] [ 9923.535266] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9923.549415] LNet: Added LNI 192.168.204.133@tcp50 [8/256/0/180] [ 9931.745597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9931.746624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9931.753038] Lustre: Skipped 2 previous similar messages [ 9931.755773] Lustre: Skipped 1 previous similar message [ 9936.866133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9936.869252] Lustre: Skipped 3 previous similar messages [ 9937.567258] Lustre: server umount lustre-MDT0000 complete [ 9938.889031] LustreError: 473235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884099 with bad export cookie 126224607787115631 [ 9938.891883] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9938.893970] LustreError: 473235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9939.017658] Lustre: server umount lustre-MDT0001 complete [ 9946.440812] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [ 9949.510205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.344290] Lustre: 471710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788884102/real 1788884102] req@00000000782e3b4f x1875780779989504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788884118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9957.353836] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9959.392181] Lustre: 471710:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788884104/real 1788884104] req@000000009eac9536 x1875780779989632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788884120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9961.440748] Lustre: 471712:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788884106/real 1788884106] req@0000000034bb9712 x1875780779989760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788884122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9964.516178] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1c0709cad106ca3 [ 9964.523535] Lustre: MGC192.168.204.133@tcp: Connection restored to 0@lo (at 0@lo) [ 9964.648226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.665113] Lustre: 471713:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788884110/real 1788884110] req@00000000954bcfe6 x1875780779990016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788884126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9966.256732] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9969.547751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9969.682530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9970.721088] Lustre: 471711:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788884115/real 1788884115] req@00000000a64c0bd6 x1875780779990144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788884131 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9971.174688] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [ 9974.069498] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9974.758395] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9976.128523] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9984.994693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.995775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9985.000170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.005737] Lustre: Skipped 1 previous similar message [ 9990.115339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9990.117900] Lustre: Skipped 1 previous similar message [ 9990.168752] Lustre: server umount lustre-OST0000 complete [ 9995.235171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9995.235953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9995.240873] Lustre: Skipped 2 previous similar messages [ 9995.241290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9995.243796] Lustre: Skipped 1 previous similar message [ 9997.784708] Lustre: server umount lustre-MDT0000 complete [ 9999.130415] LustreError: 472431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884159 with bad export cookie 126224607787117731 [ 9999.132044] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9999.135857] LustreError: 472431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9999.258150] Lustre: server umount lustre-MDT0001 complete [10003.141945] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10004.250815] Key type lgssc unregistered [10004.431584] LNet: 501367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10004.435748] LNetError: 501367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10004.439223] LNet: Removed LNI 192.168.204.133@tcp [10004.441604] LNet: Removed LNI 192.168.204.133@tcp10 [10004.443698] LNet: Removed LNI 192.168.204.133@tcp11 [10004.445681] LNet: Removed LNI 192.168.204.133@tcp12 [10004.447721] LNet: Removed LNI 192.168.204.133@tcp13 [10004.449718] LNet: Removed LNI 192.168.204.133@tcp14 [10004.451668] LNet: Removed LNI 192.168.204.133@tcp15 [10004.453864] LNet: Removed LNI 192.168.204.133@tcp16 [10004.455856] LNet: Removed LNI 192.168.204.133@tcp17 [10004.457722] LNet: Removed LNI 192.168.204.133@tcp18 [10004.459709] LNet: Removed LNI 192.168.204.133@tcp19 [10004.461742] LNet: Removed LNI 192.168.204.133@tcp20 [10004.463518] LNet: Removed LNI 192.168.204.133@tcp21 [10004.465485] LNet: Removed LNI 192.168.204.133@tcp22 [10004.466805] LNet: Removed LNI 192.168.204.133@tcp23 [10004.468306] LNet: Removed LNI 192.168.204.133@tcp24 [10004.469744] LNet: Removed LNI 192.168.204.133@tcp25 [10004.471488] LNet: Removed LNI 192.168.204.133@tcp26 [10004.472726] LNet: Removed LNI 192.168.204.133@tcp27 [10004.474225] LNet: Removed LNI 192.168.204.133@tcp28 [10004.476106] LNet: Removed LNI 192.168.204.133@tcp29 [10004.477700] LNet: Removed LNI 192.168.204.133@tcp30 [10004.479300] LNet: Removed LNI 192.168.204.133@tcp31 [10004.480929] LNet: Removed LNI 192.168.204.133@tcp32 [10004.482583] LNet: Removed LNI 192.168.204.133@tcp33 [10004.483996] LNet: Removed LNI 192.168.204.133@tcp34 [10004.485950] LNet: Removed LNI 192.168.204.133@tcp35 [10004.487475] LNet: Removed LNI 192.168.204.133@tcp36 [10004.488819] LNet: Removed LNI 192.168.204.133@tcp37 [10004.490479] LNet: Removed LNI 192.168.204.133@tcp38 [10004.491885] LNet: Removed LNI 192.168.204.133@tcp39 [10004.493563] LNet: Removed LNI 192.168.204.133@tcp40 [10004.495061] LNet: Removed LNI 192.168.204.133@tcp41 [10004.496906] LNet: Removed LNI 192.168.204.133@tcp42 [10004.498497] LNet: Removed LNI 192.168.204.133@tcp43 [10004.500241] LNet: Removed LNI 192.168.204.133@tcp44 [10004.501571] LNet: Removed LNI 192.168.204.133@tcp45 [10004.502887] LNet: Removed LNI 192.168.204.133@tcp46 [10004.504205] LNet: Removed LNI 192.168.204.133@tcp47 [10004.507097] LNet: Removed LNI 192.168.204.133@tcp48 [10004.509293] LNet: Removed LNI 192.168.204.133@tcp49 [10004.516360] LNet: Removed LNI 192.168.204.133@tcp50 [10004.818538] Key type .llcrypt unregistered [10004.819867] Key type ._llcrypt unregistered [10011.524310] Key type ._llcrypt registered [10011.525782] Key type .llcrypt registered [10011.565791] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [10017.670404] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10018.099382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10018.115563] alg: No test for adler32 (adler32-zlib) [10018.966231] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [10019.052866] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10020.640126] Key type lgssc registered [10020.966689] Lustre: Echo OBD driver; http://www.lustre.org/ [10032.689575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10032.694308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.774125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10033.784339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10033.818639] Lustre: lustre-MDT0000: new disk, initializing [10033.838349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10033.844466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10035.197233] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10039.315671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.356888] Lustre: 504533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10039.377064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10039.378802] Lustre: Skipped 1 previous similar message [10039.411665] Lustre: lustre-MDT0001: new disk, initializing [10039.460865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10039.474352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10039.477861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10040.838913] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10044.569914] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10046.413375] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10048.982936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10049.052658] Lustre: *** cfs_fail_loc=90f, val=0*** [10049.080140] Lustre: lustre-OST0000: new disk, initializing [10049.081857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10049.084748] Lustre: 506550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10049.112795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10051.017708] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10054.684310] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10055.159267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10055.163291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10055.202180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10060.262320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10060.262359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.263376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10060.265221] LustreError: Skipped 1 previous similar message [10060.269524] Lustre: Skipped 1 previous similar message [10063.125904] Lustre: server umount lustre-OST0000 complete [10065.378062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10065.378262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.378746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10065.378750] Lustre: Skipped 1 previous similar message [10065.390033] Lustre: Skipped 1 previous similar message [10070.497473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.499612] Lustre: Skipped 2 previous similar messages [10070.571417] Lustre: server umount lustre-MDT0000 complete [10071.733905] LustreError: 504527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884232 with bad export cookie 5320175811544840201 [10071.736730] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10071.737594] LustreError: 504527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10071.848092] Lustre: server umount lustre-MDT0001 complete [10075.093181] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10076.116164] Key type lgssc unregistered [10076.243504] LNet: 508455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10076.245953] LNetError: 508455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10076.253395] LNet: Removed LNI 192.168.204.133@tcp [10076.523342] Key type .llcrypt unregistered [10076.524196] Key type ._llcrypt unregistered [10083.457469] Key type ._llcrypt registered [10083.459051] Key type .llcrypt registered [10083.496898] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10083.870161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10083.904154] alg: No test for adler32 (adler32-zlib) [10084.764463] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [10084.859427] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10086.440137] Key type lgssc registered [10086.837561] Lustre: Echo OBD driver; http://www.lustre.org/ [10089.419250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10089.425479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10090.561639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10091.912117] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10094.748425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10094.878399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10096.207137] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10098.519825] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10100.289175] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10102.401947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10102.511075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10104.296123] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10106.560596] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10120.699457] loop0: detected capacity change from 0 to 204800000 [10120.761075] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10120.764290] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10120.768636] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10122.822023] loop0: detected capacity change from 0 to 204800000 [10122.942098] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10122.946205] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10122.950525] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10124.649584] loop0: detected capacity change from 0 to 204800000 [10126.567799] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.619834] Lustre: fs15246-OST0000: new disk, initializing [10126.621566] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10126.624114] Lustre: 513311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10126.648593] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10128.635190] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10132.565149] loop1: detected capacity change from 0 to 204800000 [10134.525029] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10134.568973] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10134.596568] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10134.633755] Lustre: fs15246-MDT0000: new disk, initializing [10134.692582] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10134.702029] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10134.741777] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10135.776810] Lustre: 514434:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a7db6ac0 x1875781086118144/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788884312 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10136.106397] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10143.206415] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10144.864281] LustreError: 514434:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [10144.868597] LustreError: 514434:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [10144.872096] LustreError: 514434:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [10148.324047] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10149.270185] Lustre: server umount fs15246-MDT0000 complete [10152.245379] Lustre: server umount fs15246-OST0000 complete [10156.000801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10156.003261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.007109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10158.048954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10158.052691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.058638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10161.686342] Lustre: server umount lustre-OST0000 complete [10163.681896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10163.682353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10163.685490] Lustre: Skipped 1 previous similar message [10163.687161] Lustre: Skipped 1 previous similar message [10169.044819] Lustre: server umount lustre-MDT0000 complete [10170.270331] LustreError: 509680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884331 with bad export cookie 16606392746002434265 [10170.273448] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10170.274385] LustreError: 509680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10170.410051] Lustre: server umount lustre-MDT0001 complete [10173.956438] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10174.924603] Key type lgssc unregistered [10175.046394] LNet: 517527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10175.048623] LNetError: 517527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10175.059327] LNet: Removed LNI 192.168.204.133@tcp [10175.362514] Key type .llcrypt unregistered [10175.363808] Key type ._llcrypt unregistered [10182.738185] Key type ._llcrypt registered [10182.739504] Key type .llcrypt registered [10182.780082] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10183.089666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10183.097375] alg: No test for adler32 (adler32-zlib) [10183.945855] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [10184.036785] LNet: Added LNI 192.168.204.133@tcp [8/256/0/180] [10185.624164] Key type lgssc registered [10185.954902] Lustre: Echo OBD driver; http://www.lustre.org/ [10197.696220] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [10200.882387] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10212.587128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10212.592773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.695448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10213.705717] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10213.738193] Lustre: lustre-MDT0000: new disk, initializing [10213.761129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.767853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10215.096147] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10219.156972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10219.202310] Lustre: 521958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10219.225363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10219.227571] Lustre: Skipped 1 previous similar message [10219.269974] Lustre: lustre-MDT0001: new disk, initializing [10219.302972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10219.315342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10219.318845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10220.573026] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10224.279443] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10226.074180] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10228.230112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10228.346599] Lustre: lustre-OST0000: new disk, initializing [10228.348372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10228.351185] Lustre: 523927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10228.382962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10230.352270] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_default_debug -1 all [10234.051926] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10234.360376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10234.364282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10234.398846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10235.554231] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10235.635817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10237.096816] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10237.173484] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10239.457519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10239.458517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.459913] LustreError: Skipped 1 previous similar message [10239.463929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10239.466525] Lustre: Skipped 1 previous similar message [10243.923599] Lustre: server umount lustre-OST0000 complete [10249.697795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10249.698459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10249.702364] Lustre: Skipped 2 previous similar messages [10251.293507] Lustre: server umount lustre-MDT0000 complete [10252.504133] LustreError: 521952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788884413 with bad export cookie 9934516204088156345 [10252.507565] LustreError: MGC192.168.204.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10252.508325] LustreError: 521952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10252.627206] Lustre: server umount lustre-MDT0001 complete [10260.396525] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing set_hostid [10263.557872] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing load_modules_local [10273.236682] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10002 sec ========= 12:20:32 (1788884432) [10273.735761] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:20:33 (1788884433) === [10274.753240] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:20:34 (1788884434) === [10287.289970] Lustre: DEBUG MARKER: oleg433-server.virtnet: executing unload_modules_local [10288.258331] Key type lgssc unregistered [10288.382269] LNet: 530558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10288.384706] LNetError: 530558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10288.393244] LNet: Removed LNI 192.168.204.133@tcp [10288.662235] Key type .llcrypt unregistered [10288.663704] Key type ._llcrypt unregistered