[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 525842852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003255] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008710] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010017] pid_max: default: 32768 minimum: 301 [ 0.012141] LSM: Security Framework initializing [ 0.013061] Yama: becoming mindful. [ 0.014040] SELinux: Initializing. [ 0.015085] *** VALIDATE selinux *** [ 0.024400] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028862] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032122] *** VALIDATE tmpfs *** [ 0.034469] *** VALIDATE proc *** [ 0.036090] *** VALIDATE cgroup *** [ 0.036973] *** VALIDATE cgroup2 *** [ 0.038269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039171] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041029] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.045290] debug: unmapping init [mem 0xffffffff95059000-0xffffffff95060fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056323] rcu: Hierarchical SRCU implementation. [ 0.058532] smp: Bringing up secondary CPUs ... [ 0.059607] x86: Booting SMP configuration: [ 0.060034] .... node #0, CPUs: #1 #2 #3 [ 0.063617] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152359] node 0 deferred pages initialised in 84ms [ 0.155660] devtmpfs: initialized [ 0.156261] x86/mm: Memory block size: 128MB [ 0.158862] gcov: version magic: 0x41383552 [ 0.160279] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162315] pinctrl core: initialized pinctrl subsystem [ 0.163224] [ 0.163853] ************************************************************* [ 0.164014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165014] ** ** [ 0.166016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167017] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.170013] ** your system. ** [ 0.171007] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.173011] ** kernel, report this immediately to your vendor! ** [ 0.174013] ** ** [ 0.175016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176016] ************************************************************* [ 0.177741] NET: Registered protocol family 16 [ 0.178441] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181665] cpuidle: using governor menu [ 0.184253] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185417] PCI: Using configuration type 1 for base access [ 0.186119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194073] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199101] cryptd: max_cpu_qlen set to 1000 [ 0.202267] ACPI: Added _OSI(Module Device) [ 0.204018] ACPI: Added _OSI(Processor Device) [ 0.205011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.212052] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219606] ACPI: Interpreter enabled [ 0.220078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.221016] ACPI: Using IOAPIC for interrupt routing [ 0.222135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223404] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232780] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245370] acpiphp: Slot [2] registered [ 0.247087] acpiphp: Slot [5] registered [ 0.248077] acpiphp: Slot [6] registered [ 0.249116] acpiphp: Slot [7] registered [ 0.250118] acpiphp: Slot [8] registered [ 0.252115] acpiphp: Slot [9] registered [ 0.253117] acpiphp: Slot [10] registered [ 0.255122] acpiphp: Slot [3] registered [ 0.256114] acpiphp: Slot [4] registered [ 0.258142] acpiphp: Slot [11] registered [ 0.259143] acpiphp: Slot [12] registered [ 0.261113] acpiphp: Slot [13] registered [ 0.262092] acpiphp: Slot [14] registered [ 0.264112] acpiphp: Slot [15] registered [ 0.265111] acpiphp: Slot [16] registered [ 0.267115] acpiphp: Slot [17] registered [ 0.268121] acpiphp: Slot [18] registered [ 0.270114] acpiphp: Slot [19] registered [ 0.271103] acpiphp: Slot [20] registered [ 0.272092] acpiphp: Slot [21] registered [ 0.273129] acpiphp: Slot [22] registered [ 0.274177] acpiphp: Slot [23] registered [ 0.276132] acpiphp: Slot [24] registered [ 0.278144] acpiphp: Slot [25] registered [ 0.279080] acpiphp: Slot [26] registered [ 0.280084] acpiphp: Slot [27] registered [ 0.282097] acpiphp: Slot [28] registered [ 0.283091] acpiphp: Slot [29] registered [ 0.285120] acpiphp: Slot [30] registered [ 0.286128] acpiphp: Slot [31] registered [ 0.287064] PCI host bridge to bus 0000:00 [ 0.289031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.299033] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.302044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304243] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307088] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311386] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.332022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.334019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.337020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.340132] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.342840] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348878] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.367016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379853] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424760] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.444020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506596] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.518019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.542000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.548018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.554015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.567017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.575000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.583017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.590019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.609015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.626042] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.644019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.656020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.666719] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.669443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.672451] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.674386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.676186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.680069] iommu: Default domain type: Passthrough [ 0.681342] SCSI subsystem initialized [ 0.682088] ACPI: bus type USB registered [ 0.683072] usbcore: registered new interface driver usbfs [ 0.685058] usbcore: registered new interface driver hub [ 0.686089] usbcore: registered new device driver usb [ 0.687154] pps_core: LinuxPPS API ver. 1 registered [ 0.689009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691078] PTP clock support registered [ 0.692235] EDAC MC: Ver: 3.0.0 [ 0.693432] PCI: Using ACPI for IRQ routing [ 0.694819] NetLabel: Initializing [ 0.695008] NetLabel: domain hash size = 128 [ 0.696007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.697156] NetLabel: unlabeled traffic allowed by default [ 0.699182] vgaarb: loaded [ 0.700335] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.702014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.710000] clocksource: Switched to clocksource kvm-clock [ 0.820215] VFS: Disk quotas dquot_6.6.0 [ 0.821580] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.823850] *** VALIDATE ramfs *** [ 0.824925] *** VALIDATE hugetlbfs *** [ 0.826294] pnp: PnP ACPI init [ 0.828491] pnp: PnP ACPI: found 6 devices [ 0.844367] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.847209] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.849112] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850902] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.852952] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.854939] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.857161] NET: Registered protocol family 2 [ 0.859098] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.863451] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.866274] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.870688] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.873849] TCP: Hash tables configured (established 65536 bind 65536) [ 0.876299] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.878420] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880243] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883128] NET: Registered protocol family 1 [ 0.885380] RPC: Registered named UNIX socket transport module. [ 0.887434] RPC: Registered udp transport module. [ 0.889119] RPC: Registered tcp transport module. [ 0.890545] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.892469] NET: Registered protocol family 44 [ 0.894027] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.896363] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.898673] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.900746] PCI: CLS 0 bytes, default 64 [ 0.902079] Unpacking initramfs... [ 2.348670] debug: unmapping init [mem 0xffff8de13cc54000-0xffff8de13ffbffff] [ 2.352511] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.355128] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.358196] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.863802] Initialise system trusted keyrings [ 2.865504] Key type blacklist registered [ 2.867354] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.877210] zbud: loaded [ 2.880594] *** VALIDATE nfs *** [ 2.881928] *** VALIDATE nfs4 *** [ 2.883131] pstore: using deflate compression [ 2.886719] Platform Keyring initialized [ 2.984564] NET: Registered protocol family 38 [ 2.985717] Key type asymmetric registered [ 2.986738] Asymmetric key parser 'x509' registered [ 2.988110] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.990990] io scheduler mq-deadline registered [ 2.993039] io scheduler kyber registered [ 2.994705] io scheduler bfq registered [ 2.996972] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.999725] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.002784] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.005729] ACPI: Power Button [PWRF] [ 3.011114] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.018157] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.041868] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.063373] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.117721] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.145879] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.174497] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.180123] Non-volatile memory driver v1.3 [ 3.181632] Linux agpgart interface v0.103 [ 3.217143] virtio_blk virtio1: [vda] 145944 512-byte logical blocks (74.7 MB/71.3 MiB) [ 3.219605] vda: detected capacity change from 0 to 74723328 [ 3.234431] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.236822] vdb: detected capacity change from 0 to 1073741824 [ 3.251497] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.253773] vdc: detected capacity change from 0 to 2621440000 [ 3.266343] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.268161] vdd: detected capacity change from 0 to 2621440000 [ 3.282138] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.284219] vde: detected capacity change from 0 to 4294967296 [ 3.306160] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.308284] vdf: detected capacity change from 0 to 4294967296 [ 3.327157] libphy: Fixed MDIO Bus: probed [ 3.370102] usbcore: registered new interface driver usbserial_generic [ 3.371961] usbserial: USB Serial support registered for generic [ 3.373482] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.376849] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.378260] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.380028] mousedev: PS/2 mouse device common for all mice [ 3.382104] rtc_cmos 00:05: RTC can wake from S4 [ 3.384116] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.385598] rtc_cmos 00:05: registered as rtc0 [ 3.390200] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.390216] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.392206] intel_pstate: CPU model not supported [ 3.397340] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.401627] hid: raw HID events driver (C) Jiri Kosina [ 3.403311] usbcore: registered new interface driver usbhid [ 3.404744] usbhid: USB HID core driver [ 3.405947] drop_monitor: Initializing network drop monitor service [ 3.407718] Initializing XFRM netlink socket [ 3.409369] NET: Registered protocol family 10 [ 3.411727] Segment Routing with IPv6 [ 3.412798] NET: Registered protocol family 17 [ 3.414215] mpls_gso: MPLS GSO support [ 3.419057] RAS: Correctable Errors collector initialized. [ 3.420490] AVX version of gcm_enc/dec engaged. [ 3.421624] AES CTR mode by8 optimization enabled [ 3.484420] sched_clock: Marking stable (3484402571, 0)->(4426580126, -942177555) [ 3.487799] registered taskstats version 1 [ 3.489367] Loading compiled-in X.509 certificates [ 3.491410] zswap: loaded using pool lzo/zbud [ 3.512498] Key type big_key registered [ 3.524476] Key type encrypted registered [ 3.525901] ima: No TPM chip found, activating TPM-bypass! [ 3.527588] ima: Allocated hash algorithm: sha1 [ 3.529075] ima: No architecture policies found [ 3.530292] evm: Initialising EVM extended attributes: [ 3.531777] evm: security.selinux [ 3.532682] evm: security.ima [ 3.533479] evm: security.capability [ 3.534454] evm: HMAC attrs: 0x1 [ 3.536656] rtc_cmos 00:05: setting system clock to 2026-08-14 23:51:58 UTC (1786751518) [ 3.542738] debug: unmapping init [mem 0xffffffff96003000-0xffffffff961fffff] [ 3.545317] debug: unmapping init [mem 0xffffffff94d82000-0xffffffff95058fff] [ 3.553198] Write protecting the kernel read-only data: 28672k [ 3.555792] debug: unmapping init [mem 0xffffffff93403000-0xffffffff935fffff] [ 3.558213] debug: unmapping init [mem 0xffffffff93d14000-0xffffffff93dfffff] [ 3.590485] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.597333] systemd[1]: Detected virtualization kvm. [ 3.598943] systemd[1]: Detected architecture x86-64. [ 3.600599] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.627426] systemd[1]: No hostname configured. [ 3.628768] systemd[1]: Set hostname to . [ 3.630424] random: systemd: uninitialized urandom read (16 bytes read) [ 3.632300] systemd[1]: Initializing machine ID from random generator. [ 3.695143] random: ln: uninitialized urandom read (6 bytes read) [ 3.778825] random: systemd: uninitialized urandom read (16 bytes read) [ 3.781416] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.785905] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.792775] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.391742] device-mapper: uevent: version 1.0.3 [ 4.394195] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ 4.692750] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.049711] virtio_net virtio0 ens2: renamed from eth0 [ 5.077572] scsi host0: ata_piix [ 5.084357] scsi host1: ata_piix [ 5.087637] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.091667] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.810167] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.040298] random: crng init done [ 10.041226] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.366561] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.517164] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.779523] SELinux: Disabled at runtime. [ 11.841131] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.850065] systemd[1]: Detected virtualization kvm. [ 11.852042] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.333969] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.337970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.344120] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.348445] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.352056] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.361058] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.379562] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ 12.451223] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.795437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.109508] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.158420] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.253296] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.261707] EDAC sbridge: Ver: 1.1.2 [ 14.537249] Key type dns_resolver registered [ 14.843203] NFS: Registering the id_resolver key type [ 14.845581] Key type id_resolver registered [ 14.847055] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg152-server login: [ 41.445975] libcfs: loading out-of-tree module taints kernel. [ 41.471190] Key type ._llcrypt registered [ 41.473608] Key type .llcrypt registered [ 41.523721] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 60.929533] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 62.763680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.777572] alg: No test for adler32 (adler32-zlib) [ 64.231512] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 65.051649] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 66.775431] Key type lgssc registered [ 68.811302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.772008] hrtimer: interrupt took 6015528 ns [ 89.412223] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.449074] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 155.605665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.663403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.846116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.880761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.010758] Lustre: lustre-MDT0000: new disk, initializing [ 157.205375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.259070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.329090] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.078404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.295368] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.341535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.347265] Lustre: Skipped 1 previous similar message [ 175.500502] Lustre: lustre-MDT0001: new disk, initializing [ 175.692043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 175.744194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 175.757513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.317370] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.296944] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.871914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.368070] Lustre: lustre-OST0000: new disk, initializing [ 198.383239] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.391495] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.475085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.451893] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.983134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.997775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.141545] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.067444] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.253958] Lustre: lustre-OST0001: new disk, initializing [ 224.258744] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.270894] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.376599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.897630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.905754] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.996295] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.491916] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.766487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.075613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.122337] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.219166] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing yml_node [ 265.835443] Lustre: DEBUG MARKER: Client: 2.17.57.4 [ 268.110789] Lustre: DEBUG MARKER: MDS: 2.17.57.4 [ 270.840742] Lustre: DEBUG MARKER: OSS: 2.17.57.4 [ 273.245468] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 19:56:25 EDT 2026 [ 292.130961] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.471443] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.828492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.840523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.864237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.900259] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.900980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 316.933917] Lustre: Skipped 1 previous similar message [ 316.942610] Lustre: Skipped 1 previous similar message [ 318.671113] Lustre: server umount lustre-MDT0000 complete [ 322.391611] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786751837 with bad export cookie 12760435239674813476 [ 322.394445] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.403202] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 327.136570] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.137365] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.156174] LustreError: 6523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 327.197471] Lustre: Skipped 1 previous similar message [ 327.203990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.207686] Lustre: Skipped 3 previous similar messages [ 328.843093] Lustre: server umount lustre-MDT0001 complete [ 332.662524] Lustre: server umount lustre-OST0000 complete [ 336.358811] Lustre: server umount lustre-OST0001 complete [ 344.154814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 356.421665] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 400.766638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.221407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.266316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.480676] Lustre: lustre-MDT0000: new disk, initializing [ 401.633409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.656738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.744752] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 419.100155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.219525] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.280820] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.285669] Lustre: Skipped 1 previous similar message [ 419.442924] Lustre: lustre-MDT0001: new disk, initializing [ 419.555134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 419.609053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 419.625069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.560764] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 436.575550] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 443.155203] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.707807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.973875] Lustre: lustre-OST0000: new disk, initializing [ 450.981389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.990887] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 451.119974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.592421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 452.609168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 452.741832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 458.496067] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 468.262102] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 473.782720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.969873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.879818] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 479.197426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 483.304681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.308901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.327967] LustreError: Skipped 1 previous similar message [ 483.340275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 483.360521] Lustre: Skipped 2 previous similar messages [ 483.379123] Lustre: Skipped 1 previous similar message [ 487.095950] Lustre: server umount lustre-OST0000 complete [ 493.538225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.556721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.777675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.785466] Lustre: Skipped 4 previous similar messages [ 506.336083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.665405] Lustre: server umount lustre-MDT0000 complete [ 508.900693] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.918855] LustreError: 15097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 511.142390] LustreError: 15077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752026 with bad export cookie 12760435239674815009 [ 511.178350] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.971949] Lustre: server umount lustre-MDT0001 complete [ 516.791691] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:00:29 (1786752029) [ 519.807327] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 522.094335] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:00:34 (1786752034) [ 529.749032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.262557] LustreError: 19465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.367775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 534.009514] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 535.545777] LustreError: 19466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.640912] LustreError: 19465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.135327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.667559] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 557.666682] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.809091] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 573.499984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 582.230598] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 591.366784] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 600.049863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.051592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 600.073167] Lustre: Skipped 1 previous similar message [ 600.083802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.090457] Lustre: Skipped 1 previous similar message [ 604.494957] Lustre: server umount lustre-OST0000 complete [ 614.498995] Lustre: server umount lustre-MDT0000 complete [ 615.399050] LustreError: 19461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.426312] LustreError: 19461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 618.781915] LustreError: 20200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752133 with bad export cookie 12760435239674815891 [ 618.796711] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.800714] LustreError: 20200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 619.395449] Lustre: server umount lustre-MDT0001 complete [ 635.368390] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 637.607805] Key type lgssc unregistered [ 637.859716] LNet: 23029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.882372] LNetError: 23029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.902895] LNet: Removed LNI 192.168.201.152@tcp [ 639.321276] Key type .llcrypt unregistered [ 639.323081] Key type ._llcrypt unregistered [ 666.314783] Key type ._llcrypt registered [ 666.317299] Key type .llcrypt registered [ 666.415410] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 668.074931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.183741] alg: No test for adler32 (adler32-zlib) [ 669.369423] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 669.591958] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 671.295204] Key type lgssc registered [ 672.093679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 681.698519] loop: module loaded [ 681.790436] loop0: detected capacity change from 0 to 204800000 [ 682.092530] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.103754] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 682.509224] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 688.568990] loop0: detected capacity change from 0 to 204800000 [ 688.847208] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 688.855885] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 688.864299] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.389409] loop0: detected capacity change from 0 to 204800000 [ 699.988174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 700.012581] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 701.199260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 701.238041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.339227] Lustre: lustre-MDT0000: new disk, initializing [ 701.409957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.428678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 706.002754] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 716.772946] loop1: detected capacity change from 0 to 204800000 [ 723.753821] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.017542] Lustre: lustre-OST07c6: new disk, initializing [ 724.027698] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 724.037550] Lustre: Skipped 1 previous similar message [ 724.048520] Lustre: 26219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 724.147369] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 731.841608] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 733.811840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 733.824934] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 733.868324] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 737.089417] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 737.099488] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 737.106562] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 737.111147] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 737.115620] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 737.120559] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 755.089584] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 755.100240] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.106626] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.117515] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 755.134070] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.140610] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.249697] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 760.259662] Lustre: 25208:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 760.268827] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.274252] Lustre: 25208:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.284587] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.297920] Lustre: 25208:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.308206] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 760.319882] Lustre: 25208:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.329418] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.342938] Lustre: 25208:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.352497] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.360248] Lustre: 25208:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.634706] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 763.645514] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.653809] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 763.664527] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.678040] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.693080] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.705869] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 763.720567] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.729475] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.739923] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.754435] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.762230] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.670574] Lustre: 25209:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 768.680111] Lustre: 25209:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.692031] Lustre: 25209:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.699751] Lustre: 25209:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.703991] Lustre: 25209:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.711331] Lustre: 25209:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.719467] Lustre: 25209:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 768.729448] Lustre: 25209:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.741625] Lustre: 25209:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 768.749782] Lustre: 25209:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.766755] Lustre: 25209:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.776934] Lustre: 25209:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 776.766771] Lustre: 26846:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 776.775041] Lustre: 26846:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 776.783449] Lustre: 26846:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 776.791975] Lustre: 26846:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.800488] Lustre: 26846:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 776.807486] Lustre: 26846:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.814650] Lustre: 26846:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 776.823457] Lustre: 26846:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.830830] Lustre: 26846:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 776.839994] Lustre: 26846:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 776.847493] Lustre: 26846:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 776.854076] Lustre: 26846:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 793.304392] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 793.327641] Lustre: 25207:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 793.337870] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 793.356539] Lustre: 25207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 793.369349] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.382025] Lustre: 25207:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 793.396453] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 793.409830] Lustre: 25207:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 793.421118] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 793.435956] Lustre: 25207:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 793.444714] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.457450] Lustre: 25207:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 830.945703] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 830.958159] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.982905] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 836.065701] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.192897] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.279762] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 845.474667] Lustre: server umount lustre-OST07c6 complete [ 855.578614] Lustre: server umount lustre-MDT0000 complete [ 875.187770] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 878.992469] Key type lgssc unregistered [ 879.401711] LNet: 28781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.415468] LNetError: 28781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.453595] LNet: Removed LNI 192.168.201.152@tcp [ 880.867917] Key type .llcrypt unregistered [ 880.875823] Key type ._llcrypt unregistered [ 900.803736] Key type ._llcrypt registered [ 900.810975] Key type .llcrypt registered [ 900.947151] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 901.891477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.063376] alg: No test for adler32 (adler32-zlib) [ 903.182430] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 903.435904] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 905.111210] Key type lgssc registered [ 906.542719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.002526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 916.063343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.643297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.731723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 927.214113] LustreError: 30053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.248782] LustreError: 30053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 931.822846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.448966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.562625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 945.080207] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 952.737706] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.961113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.438470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.512631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 969.944889] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 979.369885] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.383585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.385076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.401865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 998.428842] LustreError: Skipped 1 previous similar message [ 999.386977] Lustre: server umount lustre-OST0000 complete [ 1008.611809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.615436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.634295] Lustre: Skipped 2 previous similar messages [ 1008.662127] Lustre: Skipped 2 previous similar messages [ 1010.713766] Lustre: server umount lustre-MDT0000 complete [ 1013.738397] LustreError: 30054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.755856] LustreError: 30054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1015.628213] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752530 with bad export cookie 11272216162325252071 [ 1015.638546] LustreError: 30034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1015.642984] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.006901] Lustre: server umount lustre-MDT0001 complete [ 1028.999638] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1032.545208] Key type lgssc unregistered [ 1032.904996] LNet: 33794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.909252] LNetError: 33794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.923045] LNet: Removed LNI 192.168.201.152@tcp [ 1033.754500] Key type .llcrypt unregistered [ 1033.757171] Key type ._llcrypt unregistered [ 1053.074785] Key type ._llcrypt registered [ 1053.077392] Key type .llcrypt registered [ 1053.201958] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1054.783631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1054.824921] alg: No test for adler32 (adler32-zlib) [ 1055.861836] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1056.194622] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1057.879146] Key type lgssc registered [ 1059.113150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1068.148299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1068.195176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.795519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1074.936859] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1079.270748] LustreError: 35080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1079.297218] LustreError: 35080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1084.391320] LustreError: 35081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1084.441889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.891378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.740461] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1098.242076] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.339834] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.864783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1115.561301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.474581] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1134.894845] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1139.129318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1150.433133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.438563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.451811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.528669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.532894] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.923457] Lustre: server umount lustre-OST0000 complete [ 1161.697039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1161.704943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.727114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.736666] Lustre: Skipped 1 previous similar message [ 1164.769366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.770424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1164.796569] Lustre: Skipped 1 previous similar message [ 1166.159401] Lustre: server umount lustre-MDT0000 complete [ 1169.889406] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1169.935374] LustreError: 35075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1170.511661] LustreError: 35060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752685 with bad export cookie 3827845666079917073 [ 1170.526333] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.530354] LustreError: 35060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1170.761477] Lustre: server umount lustre-MDT0001 complete [ 1182.851792] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1186.433210] Key type lgssc unregistered [ 1186.866203] LNet: 38641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.878161] LNetError: 38641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.892236] LNet: Removed LNI 192.168.201.152@tcp [ 1187.966250] Key type .llcrypt unregistered [ 1187.972894] Key type ._llcrypt unregistered [ 1211.514967] Key type ._llcrypt registered [ 1211.516743] Key type .llcrypt registered [ 1211.600757] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1212.808242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.881497] alg: No test for adler32 (adler32-zlib) [ 1213.891108] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1214.059727] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1215.682159] Key type lgssc registered [ 1216.892811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.741733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.769682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.261741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.924560] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1236.454768] LustreError: 40059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.480464] LustreError: 40059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1239.494236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.872826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1244.188672] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1251.024806] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1256.074427] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1262.884836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1263.091207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.148710] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1271.285552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1277.534581] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1284.257052] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1286.115601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.125909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.159720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1286.632472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.356858] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1289.364216] Lustre: Skipped 2 previous similar messages [ 1289.654284] Lustre: server umount lustre-MDT0000 complete [ 1291.755728] LustreError: 40786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.771181] LustreError: 40786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1294.153898] LustreError: 40040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786752809 with bad export cookie 7887286381999907401 [ 1294.166973] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.478793] LustreError: 40054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.503382] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 1296.865132] LustreError: 40055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1296.869257] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.900463] Lustre: Skipped 1 previous similar message [ 1299.599844] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 1299.607686] Lustre: Skipped 1 previous similar message [ 1300.761389] Lustre: server umount lustre-MDT0001 complete [ 1322.366994] Lustre: server umount lustre-OST0000 complete [ 1340.382955] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1343.313806] Key type lgssc unregistered [ 1343.558927] LNet: 43924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.575398] LNetError: 43924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.596040] LNet: Removed LNI 192.168.201.152@tcp [ 1344.370399] Key type .llcrypt unregistered [ 1344.372206] Key type ._llcrypt unregistered [ 1368.464974] Key type ._llcrypt registered [ 1368.469403] Key type .llcrypt registered [ 1368.605892] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1369.834655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.888865] alg: No test for adler32 (adler32-zlib) [ 1371.091421] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1371.419727] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1373.255186] Key type lgssc registered [ 1374.348148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1383.699746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1383.738501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1385.337259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1389.372299] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1395.706989] LustreError: 45342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.730664] LustreError: 45342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1398.949055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1399.639369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1405.149646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1414.030801] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1422.398388] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1432.404505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.788834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.218435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1447.237210] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1456.389347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1465.843416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1465.860247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.892671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.848343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.852776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.862114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.891868] Lustre: Skipped 1 previous similar message [ 1469.118959] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 1471.754714] Lustre: server umount lustre-OST0000 complete [ 1491.430750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.459906] Lustre: Skipped 1 previous similar message [ 1491.466324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1491.487467] Lustre: Skipped 1 previous similar message [ 1497.569492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1497.586027] Lustre: Skipped 3 previous similar messages [ 1504.223901] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1504.516020] Lustre: server umount lustre-MDT0000 complete [ 1507.810542] LustreError: 47470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1507.839638] LustreError: 47470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1508.799902] LustreError: 45322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753023 with bad export cookie 9847451737665226457 [ 1508.813180] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.170902] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1523.958101] Lustre: server umount lustre-MDT0001 complete [ 1536.328532] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1539.366951] Key type lgssc unregistered [ 1539.619673] LNet: 49079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.623061] LNetError: 49079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.647863] LNet: Removed LNI 192.168.201.152@tcp [ 1540.565249] Key type .llcrypt unregistered [ 1540.567095] Key type ._llcrypt unregistered [ 1562.900198] Key type ._llcrypt registered [ 1562.901499] Key type .llcrypt registered [ 1562.986403] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1563.706628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.792038] alg: No test for adler32 (adler32-zlib) [ 1564.902661] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1565.192658] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1566.887343] Key type lgssc registered [ 1568.452947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1576.729488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1576.745452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.188600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.904361] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1588.708127] LustreError: 50495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1588.727596] LustreError: 50495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1591.271760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.626163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1596.915762] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1605.450619] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1613.068854] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1620.814950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1621.359102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.429615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1628.018529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1635.494675] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1641.119588] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1644.639737] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1656.636599] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1420h19m29s [ 1658.100333] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1662.444515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.455611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.461091] Lustre: Skipped 2 previous similar messages [ 1662.476763] Lustre: Skipped 2 previous similar messages [ 1663.522147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.527837] Lustre: Skipped 1 previous similar message [ 1666.658360] Lustre: server umount lustre-MDT0000 complete [ 1668.774968] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.079897] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1674.892352] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1674.906001] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1676.413956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.654681] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.844260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1676.914898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1677.874220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1681.660558] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1684.965513] Lustre: DEBUG MARKER: Wait for df (36782) ... [ 1686.488490] Lustre: DEBUG MARKER: done [ 1692.130042] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1692.138189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.170955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1692.177168] Lustre: Skipped 3 previous similar messages [ 1693.152122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.163435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.511205] Lustre: server umount lustre-OST0000 complete [ 1703.411755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1703.425140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.425153] Lustre: Skipped 2 previous similar messages [ 1703.425986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1703.447401] Lustre: Skipped 1 previous similar message [ 1707.515085] Lustre: server umount lustre-MDT0000 complete [ 1710.569520] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1710.587927] LustreError: 50491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1711.010148] LustreError: 50476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753225 with bad export cookie 9166881587255824905 [ 1711.015428] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.019904] LustreError: 50476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1711.288614] Lustre: server umount lustre-MDT0001 complete [ 1722.389711] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1724.976576] Key type lgssc unregistered [ 1725.348850] LNet: 55421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.352971] LNetError: 55421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.376493] LNet: Removed LNI 192.168.201.152@tcp [ 1727.255868] Key type .llcrypt unregistered [ 1727.258610] Key type ._llcrypt unregistered [ 1768.306281] Key type ._llcrypt registered [ 1768.308967] Key type .llcrypt registered [ 1768.485777] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1769.879110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1770.046932] alg: No test for adler32 (adler32-zlib) [ 1771.052135] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1771.302252] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1772.983208] Key type lgssc registered [ 1774.159658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1781.780831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1781.801416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.089334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1783.106876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1783.291310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.378630] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1795.874839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.081667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1796.118974] Lustre: 57630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1796.465930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.295117] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1808.480632] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1814.333011] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.595033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.888848] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1822.271379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.637912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1829.798485] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 1838.617238] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1846.192762] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1420h22m39s [ 1848.081474] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1855.421494] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1858.061582] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1858.165745] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1878.704306] Lustre: lustre-MDT0000: Client 5a8e2726-2064-4a97-90fe-80192165da5c (at 192.168.201.52@tcp) reconnecting [ 1898.183243] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1900.686670] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1903.194548] Lustre: DEBUG MARKER: done [ 1910.334890] Lustre: server umount lustre-OST0000 complete [ 1914.336713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1914.347062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.359633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.425932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.433417] Lustre: Skipped 2 previous similar messages [ 1921.509154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1921.524935] Lustre: Skipped 1 previous similar message [ 1927.647156] Lustre: 56953:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753426/real 1786753426] req@000000006f4d813f x1873546613677056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1927.647218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.653453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.676379] Lustre: 56953:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1927.684802] Lustre: Skipped 1 previous similar message [ 1928.159134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1928.384976] Lustre: server umount lustre-MDT0000 complete [ 1930.719136] Lustre: 56951:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753429/real 1786753429] req@000000003e820d94 x1873546613679104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1931.807213] Lustre: 56953:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753430/real 1786753430] req@00000000b4be2117 x1873546613679744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786753446 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1932.134328] LustreError: 59491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753447 with bad export cookie 8477898125979305203 [ 1932.141504] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.144059] LustreError: 59491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1932.522957] Lustre: server umount lustre-MDT0001 complete [ 1943.318817] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 1945.962267] Key type lgssc unregistered [ 1946.282409] LNet: 61965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.287286] LNetError: 61965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.308779] LNet: Removed LNI 192.168.201.152@tcp [ 1947.020930] Key type .llcrypt unregistered [ 1947.028737] Key type ._llcrypt unregistered [ 1989.762205] Key type ._llcrypt registered [ 1989.763776] Key type .llcrypt registered [ 1989.887746] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 1991.314503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1991.350853] alg: No test for adler32 (adler32-zlib) [ 1992.558993] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 1992.928616] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 1994.615759] Key type lgssc registered [ 1995.632613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1999.130596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1999.148578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.536547] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2000.565486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2000.793583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2001.860493] Lustre: Failing over lustre-MDT0000 [ 2002.059598] Lustre: server umount lustre-MDT0000 complete [ 2009.616848] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:25:22 (1786753522) [ 2016.828701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.326150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2022.407228] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2031.532381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.641194] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2031.676779] Lustre: 64757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2031.912424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.257779] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2044.851339] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2051.419580] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.804795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2059.080640] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2059.286369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.739625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2068.669871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2074.906209] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2080.939980] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2089.471524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.475173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.486701] Lustre: Skipped 1 previous similar message [ 2089.518459] Lustre: Skipped 2 previous similar messages [ 2094.567525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.586536] Lustre: Skipped 2 previous similar messages [ 2094.850725] Lustre: server umount lustre-MDT0000 complete [ 2099.825095] LustreError: 64750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753614 with bad export cookie 10102621211682148288 [ 2099.830687] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.837169] LustreError: 64750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2100.252563] Lustre: server umount lustre-MDT0001 complete [ 2104.179248] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2109.035971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2118.375620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.159510] Lustre: 63680:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753619/real 1786753619] req@000000000a0ec2aa x1873546845397632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2120.188732] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.212429] Lustre: Skipped 1 previous similar message [ 2124.323175] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8c33b8774ea418a3 [ 2124.348511] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 2124.728261] LustreError: 66603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.751347] LustreError: 66603:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2124.829902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.921242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2125.792760] Lustre: 63679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753624/real 1786753624] req@000000006d275ae6 x1873546845397760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753640 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2125.877227] LustreError: 66604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.684277] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2129.957633] LustreError: 66603:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2129.959769] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2132.065280] Lustre: 63679:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753630/real 1786753630] req@000000005ff5eb6a x1873546845398016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.095220] Lustre: 63681:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753635/real 1786753635] req@00000000c537bdb9 x1873546845398144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2136.544157] LustreError: 68529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.581120] LustreError: 68529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2138.218801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.523861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2143.722028] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2143.846830] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2152.083834] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2157.368129] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2163.502839] Lustre: server umount lustre-MDT0000 complete [ 2164.202468] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.214600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2164.227729] Lustre: Skipped 1 previous similar message [ 2164.232894] LustreError: 69504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.881177] LustreError: 64750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753681 with bad export cookie 10102621211682150563 [ 2166.883665] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2166.886472] LustreError: 64750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2167.129856] Lustre: server umount lustre-MDT0001 complete [ 2175.113983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2184.536894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.695329] Lustre: 63680:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753684/real 1786753684] req@00000000e2570ccc x1873546845423616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.744042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.779440] Lustre: Skipped 1 previous similar message [ 2191.969435] LustreError: 63677:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c16a8b2d x1873546845424768/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2192.413579] LustreError: 66604:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2192.430671] LustreError: 66604:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2192.550248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2192.641920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2196.028916] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2197.664055] Lustre: 63680:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786753696/real 1786753696] req@00000000e1f79def x1873546845424256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786753712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2197.692707] Lustre: 63680:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2204.133620] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2206.805307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.047673] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2212.341350] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2222.458905] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2228.550961] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.924376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.931864] Lustre: Skipped 1 previous similar message [ 2237.937958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.944287] Lustre: Skipped 2 previous similar messages [ 2243.045743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.051433] Lustre: Skipped 2 previous similar messages [ 2248.173147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2248.184420] Lustre: Skipped 1 previous similar message [ 2251.234282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2251.475151] Lustre: server umount lustre-MDT0000 complete [ 2253.284406] LustreError: 71211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.302161] LustreError: 71211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2256.163796] LustreError: 64748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753771 with bad export cookie 10102621211682151942 [ 2256.171971] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.175318] LustreError: 64748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2256.789893] Lustre: server umount lustre-MDT0001 complete [ 2263.403427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2266.110840] Lustre: DEBUG MARKER: files compared the same [ 2278.667987] Lustre: server umount lustre-OST0000 complete [ 2295.626116] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2298.498664] Key type lgssc unregistered [ 2298.726073] LNet: 74492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.746414] LNetError: 74492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.814752] LNet: Removed LNI 192.168.201.152@tcp [ 2300.880261] Key type .llcrypt unregistered [ 2300.884566] Key type ._llcrypt unregistered [ 2323.970594] Key type ._llcrypt registered [ 2323.972555] Key type .llcrypt registered [ 2324.060501] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2325.014465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.100872] alg: No test for adler32 (adler32-zlib) [ 2326.366414] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2326.617977] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2328.272353] Key type lgssc registered [ 2329.551671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2338.293496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2338.327807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.984195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2344.609844] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2349.542241] LustreError: 75909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.558334] LustreError: 75909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2353.650949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2353.925324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2358.148657] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2366.662486] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2372.991671] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2380.664033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2380.928297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.064932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2388.121323] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug +malloc all [ 2395.823149] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2405.855866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.863421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.875255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2408.928661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2408.928739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2408.940266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.955599] Lustre: Skipped 1 previous similar message [ 2410.552244] Lustre: server umount lustre-OST0000 complete [ 2415.597183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.602744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.615156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.167709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.169938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.286223] Lustre: server umount lustre-MDT0000 complete [ 2424.289169] LustreError: 75909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.311209] LustreError: 75909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2424.743554] LustreError: 75890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786753939 with bad export cookie 4977954899190184966 [ 2424.764891] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.052449] Lustre: server umount lustre-MDT0001 complete [ 2437.202379] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2441.195078] Key type lgssc unregistered [ 2441.544645] LNet: 79444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.553560] LNetError: 79444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.575871] LNet: Removed LNI 192.168.201.152@tcp [ 2442.479282] Key type .llcrypt unregistered [ 2442.482689] Key type ._llcrypt unregistered [ 2467.358734] Key type ._llcrypt registered [ 2467.363497] Key type .llcrypt registered [ 2467.461921] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2468.473651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2468.669361] alg: No test for adler32 (adler32-zlib) [ 2469.748906] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2470.038207] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2471.767171] Key type lgssc registered [ 2472.888183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2481.586376] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2481.603842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2497.120464] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2502.239548] LustreError: 80835:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [ 2528.927718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.907535] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2543.579333] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2552.518626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2552.731457] Lustre: *** cfs_fail_loc=706, val=0*** [ 2553.275597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.370826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2558.438285] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.461179] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2558.701398] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2563.561145] LustreError: 80860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.596726] LustreError: 80860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2568.676339] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.696237] LustreError: 80861:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2569.975087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2570.367074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2574.130079] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2582.281399] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.418284] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2591.206153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.212914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2591.220878] Lustre: Skipped 1 previous similar message [ 2591.248458] Lustre: Skipped 1 previous similar message [ 2596.324309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2596.337591] Lustre: Skipped 1 previous similar message [ 2597.170733] Lustre: server umount lustre-OST0000 complete [ 2606.049062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2606.056970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2606.077314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2607.947435] Lustre: server umount lustre-MDT0000 complete [ 2611.686439] LustreError: 81921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.710484] LustreError: 81921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2611.993330] LustreError: 80841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754126 with bad export cookie 16309477322261361522 [ 2612.004833] LustreError: 80841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2612.008856] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.426977] Lustre: server umount lustre-MDT0001 complete [ 2624.720697] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 2628.090604] Key type lgssc unregistered [ 2628.674148] LNet: 84456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.684638] LNetError: 84456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.721966] LNet: Removed LNI 192.168.201.152@tcp [ 2629.751083] Key type .llcrypt unregistered [ 2629.753459] Key type ._llcrypt unregistered [ 2655.778915] Key type ._llcrypt registered [ 2655.782711] Key type .llcrypt registered [ 2655.979620] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2657.547865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.595399] alg: No test for adler32 (adler32-zlib) [ 2658.876872] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 2659.243919] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 2661.040523] Key type lgssc registered [ 2662.917888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2673.034091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2673.070220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.762518] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2689.528157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.198830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2695.656364] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.671167] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2696.806517] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2700.769254] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.801381] LustreError: 86564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2705.889079] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.911537] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2706.580930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.072248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.547528] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.566805] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2713.312512] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2720.748980] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.765908] LustreError: 86561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2722.975846] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.990207] Lustre: MGS: Client 4e27d887-51a0-4bed-9b58-95e0cf27fb82 (at 0@lo) reconnecting [ 2723.025819] LustreError: 86575:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2723.368975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.458519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2727.540407] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2743.781249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2743.786973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2743.796615] Lustre: Skipped 1 previous similar message [ 2743.818492] Lustre: Skipped 1 previous similar message [ 2748.901501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2748.913275] Lustre: Skipped 1 previous similar message [ 2749.418466] Lustre: server umount lustre-OST0000 complete [ 2754.017063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2754.019446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2754.028020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.053316] Lustre: Skipped 1 previous similar message [ 2759.137454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.154049] Lustre: Skipped 2 previous similar messages [ 2759.741316] Lustre: server umount lustre-MDT0000 complete [ 2764.547155] Lustre: server umount lustre-MDT0001 complete [ 2768.720969] Lustre: server umount MGS complete [ 2783.173257] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:38:15 (1786754295) [ 2814.097509] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 2824.351772] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 2866.465813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.722867] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2881.010134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.337817] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2881.446528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2881.589519] Lustre: lustre-MDT0001: new disk, initializing [ 2881.743318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.792911] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2881.806980] Lustre: 93766:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2881.836736] LustreError: 93766:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2886.823551] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2898.270953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.479734] Lustre: lustre-OST0000: new disk, initializing [ 2898.483814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2898.497298] Lustre: 94531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2898.621523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.233811] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2908.769615] Lustre: 94988:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001bdf573e x1873547544798336/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786754439 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2914.372639] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2920.733855] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2920.744801] Lustre: MGS: Client 274182e9-93e8-4feb-af87-7a035798b26a (at 0@lo) reconnecting [ 2920.777293] Lustre: 93035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2920.788812] LustreError: 93752:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2920.820464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2920.889692] Lustre: lustre-MDT0000: new disk, initializing [ 2921.036270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.085715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2921.197317] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2922.144976] Lustre: 95601:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000097b51c87 x1873547544805504/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786754453 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2925.823529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 2929.172194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2929.189753] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2931.247742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2931.267649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2931.387220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2957.280890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2957.285231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.294563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2962.823725] Lustre: server umount lustre-OST0000 complete [ 2969.058682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.080892] Lustre: Skipped 1 previous similar message [ 2969.089480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.096933] Lustre: Skipped 3 previous similar messages [ 2980.319628] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2980.431931] Lustre: server umount lustre-MDT0000 complete [ 2983.395114] LustreError: 93741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.436626] LustreError: 93741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2991.409269] Lustre: server umount lustre-MDT0001 complete [ 2996.944671] Lustre: server umount MGS complete [ 3010.283499] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:42:02 (1786754522) [ 3031.920323] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3035.282763] Key type lgssc unregistered [ 3035.544565] LNet: 98109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.550891] LNetError: 98109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.565795] LNet: Removed LNI 192.168.201.152@tcp [ 3036.303549] Key type .llcrypt unregistered [ 3036.305796] Key type ._llcrypt unregistered [ 3053.597831] Key type ._llcrypt registered [ 3053.603424] Key type .llcrypt registered [ 3053.704132] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3054.911725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3054.932514] alg: No test for adler32 (adler32-zlib) [ 3056.031776] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3056.317060] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3058.095480] Key type lgssc registered [ 3059.357456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.002515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.030263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.562933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.245405] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3080.678484] LustreError: 99337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3080.701703] LustreError: 99337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3085.099383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.550988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3090.147938] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3098.556652] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3104.645639] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3112.737297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.159261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.221160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3119.085161] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3128.085660] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3142.121134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3142.123869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.138196] LustreError: Skipped 1 previous similar message [ 3142.140976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3142.158591] Lustre: Skipped 1 previous similar message [ 3142.194409] Lustre: Skipped 1 previous similar message [ 3147.238518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3147.251837] Lustre: Skipped 1 previous similar message [ 3148.157477] Lustre: server umount lustre-OST0000 complete [ 3157.485890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.514339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.517152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3157.525942] Lustre: Skipped 1 previous similar message [ 3157.569123] Lustre: Skipped 1 previous similar message [ 3159.772638] Lustre: server umount lustre-MDT0000 complete [ 3162.595897] LustreError: 101084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.621519] LustreError: 101084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3163.665323] LustreError: 99318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754678 with bad export cookie 5785586099541762080 [ 3163.668078] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.677018] LustreError: 99318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3164.110560] Lustre: server umount lustre-MDT0001 complete [ 3176.238945] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3179.975805] Key type lgssc unregistered [ 3180.176784] LNet: 102898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3180.190650] LNetError: 102898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3180.240729] LNet: Removed LNI 192.168.201.152@tcp [ 3181.661327] Key type .llcrypt unregistered [ 3181.667305] Key type ._llcrypt unregistered [ 3201.689659] Key type ._llcrypt registered [ 3201.692769] Key type .llcrypt registered [ 3201.888864] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3203.409303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3203.468089] alg: No test for adler32 (adler32-zlib) [ 3204.528626] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3204.795242] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3206.503191] Key type lgssc registered [ 3207.631150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3214.367399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3214.374088] LustreError: 103982:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3214.376899] LustreError: 103985:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3214.383264] LustreError: 103982:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 3214.390371] LustreError: 103982:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3214.396764] LustreError: 103982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3214.406179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.078254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.635593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.978558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3229.341506] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3235.460408] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lsmod [ 3238.807035] LustreError: 105192:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3238.809262] LustreError: 105193:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3238.832467] LustreError: 105192:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3238.839289] LustreError: 105192:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3238.856569] LustreError: 105192:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3238.859579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3239.407309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3244.519591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.536270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.547625] Lustre: Skipped 2 previous similar messages [ 3248.592930] Lustre: server umount lustre-MDT0000 complete [ 3249.636183] LustreError: 104019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3249.646890] LustreError: 104019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3250.658624] LustreError: 104456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3250.680905] LustreError: 104456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3252.285656] LustreError: 103999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754767 with bad export cookie 16867648071276309404 [ 3252.286626] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.300266] LustreError: 103999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3252.743930] LustreError: 104453:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006671dcd3 x1873548116250368/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3252.950207] Lustre: server umount lustre-MDT0001 complete [ 3266.588990] Lustre: server umount lustre-OST0000 complete [ 3278.885455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3279.491865] LustreError: 106258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.587839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3284.975053] LustreError: 106259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.684381] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3289.058674] LustreError: 106258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3294.889400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3295.246287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3300.083699] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3309.015826] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3314.637240] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3320.576896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3320.741246] LustreError: 108077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.757542] LustreError: 108077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3320.882297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.990359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3327.888626] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3336.587233] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3345.377670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.388242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3345.406389] Lustre: Skipped 3 previous similar messages [ 3345.424980] Lustre: Skipped 1 previous similar message [ 3350.513654] LustreError: 108077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.537304] LustreError: 108077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3350.630503] Lustre: server umount lustre-OST0000 complete [ 3355.617964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3355.622334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3355.642022] Lustre: Skipped 1 previous similar message [ 3360.740423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.761784] Lustre: Skipped 2 previous similar messages [ 3361.867092] Lustre: server umount lustre-MDT0000 complete [ 3365.784119] LustreError: 106240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786754880 with bad export cookie 16867648071276310104 [ 3365.792595] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3365.797210] LustreError: 106240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3366.253164] Lustre: server umount lustre-MDT0001 complete [ 3377.536439] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3380.432896] Key type lgssc unregistered [ 3380.812517] LNet: 109793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3380.818435] LNetError: 109793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3380.842801] LNet: Removed LNI 192.168.201.152@tcp [ 3381.825831] Key type .llcrypt unregistered [ 3381.828704] Key type ._llcrypt unregistered [ 3408.967914] Key type ._llcrypt registered [ 3408.971824] Key type .llcrypt registered [ 3409.102940] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3410.177862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3410.325544] alg: No test for adler32 (adler32-zlib) [ 3411.326369] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3411.510045] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3413.169584] Key type lgssc registered [ 3414.212785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3421.777206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3421.797506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.263939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3427.373767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3433.451336] LustreError: 111213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.501522] LustreError: 111213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3436.223891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.527074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3440.802781] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3451.312610] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3460.985375] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3468.546209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.877134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3476.592909] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3483.814341] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3493.550709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3499.660129] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3502.562680] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3508.205805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.219310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.234129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.217805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3509.221274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3513.294138] Lustre: server umount lustre-OST0000 complete [ 3518.435430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3518.446306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3518.459528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3518.464456] Lustre: Skipped 1 previous similar message [ 3522.970338] Lustre: server umount lustre-MDT0000 complete [ 3524.577978] LustreError: 112227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.599584] LustreError: 112227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3526.309898] LustreError: 111193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755041 with bad export cookie 6159296693958360685 [ 3526.320462] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3526.628521] Lustre: server umount lustre-MDT0001 complete [ 3539.000957] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3541.705342] Key type lgssc unregistered [ 3541.992813] LNet: 114876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.997647] LNetError: 114876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3542.020594] LNet: Removed LNI 192.168.201.152@tcp [ 3542.774547] Key type .llcrypt unregistered [ 3542.776549] Key type ._llcrypt unregistered [ 3561.127801] Key type ._llcrypt registered [ 3561.132567] Key type .llcrypt registered [ 3561.287341] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3562.185138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.289197] alg: No test for adler32 (adler32-zlib) [ 3563.301292] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3563.492125] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3565.135379] Key type lgssc registered [ 3566.075571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3575.529948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3575.555733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.124565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.262952] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3587.559718] LustreError: 116107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.575617] LustreError: 116107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3589.053051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3589.330352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3592.947896] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3600.506663] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3606.014988] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3612.354444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.717151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3619.105104] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3623.061644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3627.859681] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3637.740787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3637.751335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.760434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.753330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3638.757279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3642.602425] Lustre: server umount lustre-OST0000 complete [ 3648.995766] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.006100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3649.015529] Lustre: Skipped 2 previous similar messages [ 3652.409447] Lustre: server umount lustre-MDT0000 complete [ 3654.122190] LustreError: 116107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.140660] LustreError: 116107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3656.208565] LustreError: 116088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755171 with bad export cookie 14838874463365194781 [ 3656.222918] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3656.224034] LustreError: 116088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3656.648540] Lustre: server umount lustre-MDT0001 complete [ 3666.925561] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3670.371251] Key type lgssc unregistered [ 3670.762453] LNet: 119643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3670.769069] LNetError: 119643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3670.792794] LNet: Removed LNI 192.168.201.152@tcp [ 3671.824499] Key type .llcrypt unregistered [ 3671.827145] Key type ._llcrypt unregistered [ 3696.823824] Key type ._llcrypt registered [ 3696.829215] Key type .llcrypt registered [ 3696.963884] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3698.448749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3698.489293] alg: No test for adler32 (adler32-zlib) [ 3699.656525] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3700.009319] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3701.655170] Key type lgssc registered [ 3702.811246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3710.345964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3710.376683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.755492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.818298] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3721.199827] LustreError: 121068:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3721.223711] LustreError: 121068:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3724.361168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.809342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.206723] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3736.979276] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3742.903286] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3750.600991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3751.171704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.295955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3757.739911] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3765.206092] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.908732] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3779.463859] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3779.466875] Lustre: Skipped 2 previous similar messages [ 3783.067054] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3783.071907] Lustre: Skipped 2 previous similar messages [ 3784.815527] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3784.847908] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3790.099392] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3790.109253] Lustre: Skipped 2 previous similar messages [ 3796.021606] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.52@tcp 192.168.0.[2,10]@tcp [ 3803.438597] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.52/24@tcp in log lustre-MDT0000 [ 3803.446403] Lustre: Skipped 2 previous similar messages [ 3812.961026] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 3812.968309] Lustre: Skipped 1 previous similar message [ 3821.026352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3821.042172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.053680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.063835] Lustre: Skipped 1 previous similar message [ 3821.077764] Lustre: Skipped 1 previous similar message [ 3821.857378] Lustre: server umount lustre-OST0000 complete [ 3826.146310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.148591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3826.160441] Lustre: Skipped 1 previous similar message [ 3826.174705] Lustre: Skipped 1 previous similar message [ 3831.266319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.278783] Lustre: Skipped 1 previous similar message [ 3832.147979] Lustre: server umount lustre-MDT0000 complete [ 3836.356551] LustreError: 121050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755351 with bad export cookie 17569380367081625843 [ 3836.359106] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.365996] LustreError: 121050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3836.760036] Lustre: server umount lustre-MDT0001 complete [ 3848.355245] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 3850.743272] Key type lgssc unregistered [ 3851.127068] LNet: 126051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3851.131653] LNetError: 126051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.158962] LNet: Removed LNI 192.168.201.152@tcp [ 3851.929626] Key type .llcrypt unregistered [ 3851.932329] Key type ._llcrypt unregistered [ 3881.327674] Key type ._llcrypt registered [ 3881.331741] Key type .llcrypt registered [ 3881.427976] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 3882.420323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3882.545261] alg: No test for adler32 (adler32-zlib) [ 3883.722802] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 3884.018300] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 3885.743333] Key type lgssc registered [ 3886.966677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3894.708230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3894.725079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3896.390957] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3896.412104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3896.437756] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.52/24@tcp [ 3901.349978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3906.530258] LustreError: 127615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3906.564745] LustreError: 127615:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3910.564888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.791893] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3910.835410] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.52/24@tcp [ 3910.861975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3914.854317] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3922.215192] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3928.592973] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3936.796368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3937.321215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.461305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3944.097497] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 3951.985365] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3971.552520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.556360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3971.576186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3971.607881] Lustre: Skipped 1 previous similar message [ 3973.671786] Lustre: server umount lustre-OST0000 complete [ 3981.792781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.814420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.822902] Lustre: Skipped 2 previous similar messages [ 3984.899415] Lustre: server umount lustre-MDT0000 complete [ 3986.913329] LustreError: 128687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.934065] LustreError: 128687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3988.818179] LustreError: 127593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755503 with bad export cookie 2094842983668759358 [ 3988.821797] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.840405] LustreError: 127593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3989.168980] Lustre: server umount lustre-MDT0001 complete [ 4002.926853] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4006.207943] Key type lgssc unregistered [ 4006.523755] LNet: 131206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4006.527499] LNetError: 131206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4006.538298] LNet: Removed LNI 192.168.201.152@tcp [ 4007.524513] Key type .llcrypt unregistered [ 4007.526561] Key type ._llcrypt unregistered [ 4040.382985] Key type ._llcrypt registered [ 4040.386110] Key type .llcrypt registered [ 4040.531560] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4057.772743] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4058.973313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4059.194338] alg: No test for adler32 (adler32-zlib) [ 4060.384568] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 4060.718120] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4062.431161] Key type lgssc registered [ 4063.936949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4102.589565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4102.624636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.919815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4103.942867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4104.016481] Lustre: lustre-MDT0000: new disk, initializing [ 4104.108863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.146780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4109.095492] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4121.058318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.183668] Lustre: 134614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4121.231233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4121.235112] Lustre: Skipped 1 previous similar message [ 4121.320509] Lustre: lustre-MDT0001: new disk, initializing [ 4121.436569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4121.481638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4121.496809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4126.485856] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4138.458938] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4144.790651] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4153.355832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.913988] Lustre: lustre-OST0000: new disk, initializing [ 4153.922092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4153.927324] Lustre: 136593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4154.069724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.634717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4160.645800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4160.772276] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4160.838697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4170.968614] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.832858] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4176.100309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4180.560740] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4180.723467] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4186.596129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.596661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.601257] LustreError: Skipped 1 previous similar message [ 4186.604173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.630233] Lustre: Skipped 1 previous similar message [ 4186.643226] Lustre: Skipped 1 previous similar message [ 4188.878966] Lustre: server umount lustre-OST0000 complete [ 4193.249285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4193.255238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.284832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.833720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.835583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.864665] Lustre: Skipped 1 previous similar message [ 4199.479969] Lustre: server umount lustre-MDT0000 complete [ 4203.886406] LustreError: 134605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755718 with bad export cookie 12265657732029974460 [ 4203.899686] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4203.906462] LustreError: 134605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4204.542298] Lustre: server umount lustre-MDT0001 complete [ 4216.176074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4216.674582] LustreError: 138810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.706129] LustreError: 138810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4216.800453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.921823] LustreError: 138811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4222.215878] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4227.041777] LustreError: 138810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4231.906755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.270051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4236.642646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4245.550776] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4252.565259] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4260.076661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.662567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.395918] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4274.501587] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4279.591218] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4279.795652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4283.916908] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4284.107618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4292.770747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4292.913931] Lustre: lustre-OST0001: new disk, initializing [ 4292.916930] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4292.920949] Lustre: 142181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4293.012068] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4299.334852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4301.430784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4301.444134] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4301.571126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4307.550737] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4307.769160] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4312.675625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4312.958725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4319.282248] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92f484e68800.ost_server_uuid 50 [ 4321.308734] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92f484e68800.ost_server_uuid in FULL state after 0 sec [ 4332.530840] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.534964] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.541554] LustreError: Skipped 1 previous similar message [ 4332.546290] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4332.565867] Lustre: Skipped 1 previous similar message [ 4337.416828] Lustre: server umount lustre-OST0001 complete [ 4337.666600] LustreError: 141745:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.703543] LustreError: 141745:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4342.767933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.773331] LustreError: 140632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.774423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4342.774428] Lustre: Skipped 1 previous similar message [ 4342.793323] Lustre: Skipped 1 previous similar message [ 4342.815445] LustreError: 140632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4347.479342] Lustre: server umount lustre-OST0000 complete [ 4352.999396] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.012824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4353.016473] Lustre: Skipped 2 previous similar messages [ 4365.279168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4365.469730] Lustre: server umount lustre-MDT0000 complete [ 4367.333775] LustreError: 138810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4367.350601] LustreError: 138810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4369.443874] LustreError: 138791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755884 with bad export cookie 12265657732029975342 [ 4369.457131] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4369.996063] Lustre: server umount lustre-MDT0001 complete [ 4390.161820] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4393.624506] Key type lgssc unregistered [ 4393.983820] LNet: 145190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4393.992650] LNetError: 145190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4394.009056] LNet: Removed LNI 192.168.201.152@tcp [ 4395.083443] Key type .llcrypt unregistered [ 4395.086742] Key type ._llcrypt unregistered [ 4439.992810] Key type ._llcrypt registered [ 4439.995407] Key type .llcrypt registered [ 4440.098805] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4457.247456] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4459.059058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4459.081040] alg: No test for adler32 (adler32-zlib) [ 4460.149298] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 4460.450551] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4462.127263] Key type lgssc registered [ 4463.236017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4496.847741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4496.868842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.197559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4498.219351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4498.322553] Lustre: lustre-MDT0000: new disk, initializing [ 4498.370666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.385986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4503.679968] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4514.255230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.418589] Lustre: 149348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4514.477295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4514.481569] Lustre: Skipped 1 previous similar message [ 4514.581216] Lustre: lustre-MDT0001: new disk, initializing [ 4514.689434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4514.740397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4514.748817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4519.359538] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4529.165693] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4535.458675] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4543.369613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4543.896541] Lustre: lustre-OST0000: new disk, initializing [ 4543.902496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4543.909190] Lustre: 151317:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4544.042753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.948825] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4553.278400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4553.283159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4553.396474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4562.709942] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.820134] Lustre: Failing over lustre-OST0000 [ 4568.075969] Lustre: server umount lustre-OST0000 complete [ 4585.311410] Lustre: 147570:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756084/real 1786756084] req@00000000a485eaad x1873549433276288/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786756100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4585.338097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.399750] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756084/real 1786756084] req@00000000c89d8b05 x1873549433275648/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786756100 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4586.437876] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4588.752956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4589.083359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.110620] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4590.121725] Lustre: 147573:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756089/real 1786756089] req@00000000008369f6 x1873549433277568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786756105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4595.236741] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4596.002640] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4596.191197] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756094/real 1786756094] req@0000000087fd71ad x1873549433278720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786756110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4596.218198] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4596.409712] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4596.411752] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4600.313738] Lustre: Failing over lustre-MDT0000 [ 4600.663322] Lustre: server umount lustre-MDT0000 complete [ 4601.376811] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756099/real 1786756099] req@00000000c91041c0 x1873549433279872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786756115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4601.426183] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4601.830173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4601.845097] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4601.859444] Lustre: Skipped 1 previous similar message [ 4605.411422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.443478] Lustre: Skipped 1 previous similar message [ 4606.601268] LustreError: 149354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.630883] LustreError: 149354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4610.529466] LustreError: 149355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.544872] LustreError: 149355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4611.732341] LustreError: 149355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.652745] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.675221] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4620.770390] LustreError: 149355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.798539] LustreError: 149355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4621.072087] Lustre: 147571:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786756120/real 1786756120] req@00000000311cbc1e x1873549433291264/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786756136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4621.091503] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.268169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.519977] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4631.527952] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6ab397d081b0fb44 [ 4631.555940] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4631.582973] Lustre: MGC192.168.201.152@tcp: Connection restored to 0@lo (at 0@lo) [ 4631.588190] Lustre: Skipped 1 previous similar message [ 4632.076153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4632.166428] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4633.224824] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4637.175954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4637.207709] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4637.261694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4638.608077] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4647.394785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.408601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4647.410816] Lustre: Skipped 1 previous similar message [ 4647.430118] Lustre: Skipped 1 previous similar message [ 4649.345949] Lustre: server umount lustre-OST0000 complete [ 4657.633101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4657.636023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.645532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4657.666944] Lustre: Skipped 1 previous similar message [ 4660.584888] Lustre: server umount lustre-MDT0000 complete [ 4662.764927] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4662.787309] LustreError: 149358:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4664.651955] LustreError: 149338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756179 with bad export cookie 7688655910636419908 [ 4664.660295] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4664.664749] LustreError: 149338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4665.041329] Lustre: server umount lustre-MDT0001 complete [ 4682.010869] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 4686.377531] Key type lgssc unregistered [ 4686.681733] LNet: 154978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4686.687776] LNetError: 154978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4686.706388] LNet: Removed LNI 192.168.201.152@tcp [ 4687.576648] Key type .llcrypt unregistered [ 4687.580256] Key type ._llcrypt unregistered [ 4726.191668] Key type ._llcrypt registered [ 4726.194637] Key type .llcrypt registered [ 4726.299663] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4727.286344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4727.321190] alg: No test for adler32 (adler32-zlib) [ 4728.620906] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 4728.895238] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 4730.567233] Key type lgssc registered [ 4731.881529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4739.439970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4739.465074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.020592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.723190] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4750.305783] LustreError: 156513:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.320790] LustreError: 156513:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4753.525566] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4755.427947] LustreError: 156514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4757.198830] LustreError: 156542:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4757.206215] LustreError: 156542:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4757.211323] LustreError: 156542:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 4757.457990] Lustre: server umount lustre-MDT0000 complete [ 4772.020227] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 4781.399593] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 4815.814390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.147244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.181838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4816.296497] Lustre: lustre-MDT0000: new disk, initializing [ 4816.382350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4816.411954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4820.607492] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4830.734436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4830.852127] Lustre: 159847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4830.953511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4830.956533] Lustre: Skipped 1 previous similar message [ 4831.012729] Lustre: lustre-MDT0001: new disk, initializing [ 4831.132179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4831.184091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4831.200949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4835.603057] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4845.581311] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4851.501810] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4859.951569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.356802] Lustre: lustre-OST0000: new disk, initializing [ 4860.367590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4860.377160] Lustre: 161815:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4860.559221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4866.696815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4866.732125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4866.891288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4868.574720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 4879.782975] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4884.868213] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4886.536730] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:13:19 (1786756399) [ 5148.640349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5148.642590] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.657328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5148.692261] Lustre: Skipped 1 previous similar message [ 5150.605516] Lustre: server umount lustre-MDT0000 complete [ 5153.761437] LustreError: 160724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5153.792224] LustreError: 160724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5154.072233] LustreError: 159839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756669 with bad export cookie 3275545969399471186 [ 5154.079779] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5154.081514] LustreError: 159839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5158.880711] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5158.883635] LustreError: 160725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.887500] Lustre: Skipped 1 previous similar message [ 5158.887980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5158.887984] Lustre: Skipped 2 previous similar messages [ 5159.904485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5160.510199] Lustre: server umount lustre-MDT0001 complete [ 5171.439967] Lustre: server umount lustre-OST0000 complete [ 5181.898625] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5193.272677] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5233.187941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.624966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5233.654550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5233.808511] Lustre: lustre-MDT0000: new disk, initializing [ 5233.918672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5233.967476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5240.128554] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5252.600469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5252.814470] Lustre: 166480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5252.901062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5252.907495] Lustre: Skipped 1 previous similar message [ 5253.060549] Lustre: lustre-MDT0001: new disk, initializing [ 5253.203393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5253.261111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5253.279559] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5258.326894] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5271.510907] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5280.537751] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5288.145277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5288.432790] Lustre: lustre-OST0000: new disk, initializing [ 5288.439849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5288.447380] Lustre: 168454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5288.528525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.801115] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5297.273090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5297.291774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5297.462505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5307.013114] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5312.190985] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5312.563225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5317.204267] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5317.529578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5323.235582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5323.236696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.256913] LustreError: Skipped 1 previous similar message [ 5323.259649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5323.274056] Lustre: Skipped 1 previous similar message [ 5323.301383] Lustre: Skipped 1 previous similar message [ 5325.940968] Lustre: server umount lustre-OST0000 complete [ 5333.481484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.528359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5333.545562] Lustre: Skipped 1 previous similar message [ 5337.176896] Lustre: server umount lustre-MDT0000 complete [ 5338.593338] LustreError: 166493:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.647258] LustreError: 166493:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5342.901883] LustreError: 168453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756857 with bad export cookie 3275545969399630016 [ 5342.909230] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5342.920796] LustreError: 168453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5343.635604] Lustre: server umount lustre-MDT0001 complete [ 5353.155463] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:21:06 (1786756866) [ 5369.995344] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5382.360569] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5430.170930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5430.460210] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5430.490262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5430.672152] Lustre: lustre-MDT0000: new disk, initializing [ 5430.782270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.806575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5437.513291] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5450.940702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.116583] Lustre: 173214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5451.425736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5455.932285] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5467.879812] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5476.044890] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5483.504598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5483.878350] Lustre: lustre-OST0000: new disk, initializing [ 5483.880540] Lustre: Skipped 1 previous similar message [ 5483.883733] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5483.887661] Lustre: Skipped 2 previous similar messages [ 5483.892391] Lustre: 175191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5489.818294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5489.839956] Lustre: Skipped 1 previous similar message [ 5489.842585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5490.008229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5490.962693] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5501.697492] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5510.630158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5510.630644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.635117] LustreError: Skipped 1 previous similar message [ 5510.639445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5510.653722] Lustre: Skipped 2 previous similar messages [ 5515.000550] Lustre: server umount lustre-OST0000 complete [ 5520.865897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5520.875971] Lustre: Skipped 1 previous similar message [ 5525.682458] Lustre: server umount lustre-MDT0000 complete [ 5525.985905] LustreError: 173220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.003647] LustreError: 173220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5529.952852] LustreError: 173204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757044 with bad export cookie 3275545969399630898 [ 5529.963983] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5529.967126] LustreError: 173204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5530.311837] Lustre: server umount lustre-MDT0001 complete [ 5540.549081] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:24:13 (1786757053) [ 5560.250295] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 5572.323268] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5616.245648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.599830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.668682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5616.814841] Lustre: lustre-MDT0000: new disk, initializing [ 5616.912511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5616.919459] Lustre: Skipped 2 previous similar messages [ 5616.942736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5621.811105] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5632.753203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.897411] Lustre: 179515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5633.192070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5638.695767] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5652.339214] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5661.324166] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5669.443624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5669.761138] Lustre: 181495:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5676.124523] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5676.312737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5676.724324] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5687.718607] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5702.121019] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5702.127542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5702.129259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5702.129265] Lustre: Skipped 3 previous similar messages [ 5702.175962] Lustre: Skipped 1 previous similar message [ 5703.311733] Lustre: server umount lustre-OST0000 complete [ 5709.799906] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5709.813633] LustreError: Skipped 1 previous similar message [ 5714.513264] Lustre: server umount lustre-MDT0000 complete [ 5717.473186] LustreError: 179524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5717.503314] LustreError: 179524:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5718.648830] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757233 with bad export cookie 3275545969399631983 [ 5718.660305] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5718.663487] LustreError: 179508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5718.933808] Lustre: server umount lustre-MDT0001 complete [ 5731.382956] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5734.289978] Key type lgssc unregistered [ 5734.664358] LNet: 183354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5734.669754] LNetError: 183354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5734.688351] LNet: Removed LNI 192.168.201.152@tcp [ 5735.787118] Key type .llcrypt unregistered [ 5735.788762] Key type ._llcrypt unregistered [ 5761.320384] Key type ._llcrypt registered [ 5761.324727] Key type .llcrypt registered [ 5761.501547] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5763.112212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5763.150679] alg: No test for adler32 (adler32-zlib) [ 5764.382589] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5764.654503] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5766.407159] Key type lgssc registered [ 5768.520700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5778.942681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5778.972696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.582404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.574308] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5790.701722] LustreError: 184775:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5790.735157] LustreError: 184775:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5795.078455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.535663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5800.359552] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5808.843020] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5816.983637] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5825.939538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5826.493293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.513697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5833.090840] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5842.583674] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5862.370348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5862.386494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.392511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5862.414692] Lustre: Skipped 1 previous similar message [ 5867.240168] Lustre: server umount lustre-OST0000 complete [ 5872.609587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.613601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.623476] Lustre: Skipped 1 previous similar message [ 5872.632530] Lustre: Skipped 2 previous similar messages [ 5877.728166] LustreError: 185504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.736246] LustreError: 185504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5877.795644] Lustre: server umount lustre-MDT0000 complete [ 5882.495361] LustreError: 184758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757397 with bad export cookie 6319728476248238493 [ 5882.499985] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5882.507524] LustreError: 184758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5883.251787] Lustre: server umount lustre-MDT0001 complete [ 5896.506717] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 5898.776629] Key type lgssc unregistered [ 5899.012299] LNet: 188335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5899.024826] LNetError: 188335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5899.042638] LNet: Removed LNI 192.168.201.152@tcp [ 5899.910322] Key type .llcrypt unregistered [ 5899.913251] Key type ._llcrypt unregistered [ 5927.000598] Key type ._llcrypt registered [ 5927.005832] Key type .llcrypt registered [ 5927.240703] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 5929.200928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.257720] alg: No test for adler32 (adler32-zlib) [ 5930.419697] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 5930.678330] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 5932.369740] Key type lgssc registered [ 5933.637381] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5942.715647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5942.747730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5944.316873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.463421] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5954.538080] LustreError: 189751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5954.568886] LustreError: 189751:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5959.655131] LustreError: 189750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.246244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5961.706725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5968.190279] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 5977.362343] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5983.799399] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5992.347482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.821649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.616460] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6008.526398] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6014.328190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6021.601330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.610475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.630600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.672882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6024.696591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6024.728706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6025.899108] Lustre: lustre-OST0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 6025.901631] Lustre: Skipped 1 previous similar message [ 6027.772708] Lustre: server umount lustre-OST0000 complete [ 6036.807860] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92f488f73800.ost_server_uuid 50 [ 6038.890116] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6053.856942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6053.863129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6053.870136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.456728] Lustre: server umount lustre-MDT0000 complete [ 6059.489150] LustreError: 189750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6059.516606] LustreError: 189750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6062.879082] LustreError: 190485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757577 with bad export cookie 4089412313557380396 [ 6062.882729] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.888446] LustreError: 190485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6063.164722] Lustre: server umount lustre-MDT0001 complete [ 6072.258469] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:33:05 (1786757585) [ 6078.716961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.147585] LustreError: 193683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.249210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.667441] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6084.576718] LustreError: 193684:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.485308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6092.857399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.775347] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6105.402731] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6113.173924] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6120.403433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.052091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.115568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6129.536526] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6139.839457] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6148.088788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6148.391915] Lustre: lustre-OST0001: new disk, initializing [ 6148.397549] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6148.415725] Lustre: 196528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6148.589666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6153.844386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6153.856628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6153.978785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6155.326345] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6167.805616] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6171.110206] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6171.125198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.143276] Lustre: Skipped 1 previous similar message [ 6171.154877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.165663] Lustre: Skipped 2 previous similar messages [ 6177.351975] Lustre: server umount lustre-OST0000 complete [ 6178.963762] LustreError: 196112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.993476] LustreError: 196112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6185.706646] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6185.861306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6191.702786] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6192.180465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6195.178685] LustreError: 195504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.220417] LustreError: 195504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6205.410965] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6205.425789] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6205.428033] Lustre: Skipped 2 previous similar messages [ 6205.433396] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6205.436667] Lustre: Skipped 5 previous similar messages [ 6205.452497] LustreError: Skipped 1 previous similar message [ 6211.287882] Lustre: server umount lustre-OST0001 complete [ 6215.656918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6215.659037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6221.814432] Lustre: server umount lustre-MDT0000 complete [ 6226.632794] LustreError: 195527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757741 with bad export cookie 4089412313557381467 [ 6226.642298] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.664162] LustreError: 195527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6227.469559] Lustre: server umount lustre-MDT0001 complete [ 6254.500468] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:36:07 (1786757767) [ 6262.313631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.917546] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6262.990534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6263.491517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6269.811774] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6280.546185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.631263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6280.690064] Lustre: 199967:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6286.584470] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6296.505472] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6302.957745] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6311.424392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6311.758041] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6319.818569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6322.928586] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6333.627858] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6341.448709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.585682] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6341.935926] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.938494] Lustre: Skipped 2 previous similar messages [ 6348.006599] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6350.107492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6356.566807] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6360.546654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6360.556324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6360.562784] Lustre: Skipped 2 previous similar messages [ 6360.566861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6360.575332] Lustre: Skipped 6 previous similar messages [ 6365.511878] Lustre: server umount lustre-OST0000 complete [ 6365.667649] LustreError: 201818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6365.674333] LustreError: 201818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6384.486980] Lustre: server umount lustre-OST0001 complete [ 6396.391763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6396.401731] Lustre: Skipped 5 previous similar messages [ 6399.597201] Lustre: server umount lustre-MDT0000 complete [ 6402.528309] Lustre: 189063:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786757901/real 1786757901] req@00000000b7807f6d x1873550974818944/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786757917 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6402.587488] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.605684] Lustre: Skipped 2 previous similar messages [ 6403.925879] LustreError: 199962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757918 with bad export cookie 4089412313557382874 [ 6403.927573] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6403.936077] LustreError: 199962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6404.654876] Lustre: server umount lustre-MDT0001 complete [ 6435.665599] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:39:08 (1786757948) [ 6454.230429] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 6465.311318] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 6508.292774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.608607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6508.650894] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6508.784872] Lustre: lustre-MDT0000: new disk, initializing [ 6508.921316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.960218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6513.451399] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6525.450225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.583304] Lustre: 208072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6525.640631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6525.645320] Lustre: Skipped 1 previous similar message [ 6525.763746] Lustre: lustre-MDT0001: new disk, initializing [ 6525.906832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6525.918945] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6530.775435] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6540.680178] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6547.556399] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6554.744539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.067920] Lustre: lustre-OST0000: new disk, initializing [ 6555.072580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6555.078568] Lustre: 210041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6562.812874] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6562.941342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6562.957729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6563.172536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6572.689762] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6578.409643] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6578.589962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6584.305996] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6584.594611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6588.896261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.908325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.918630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6588.921270] Lustre: Skipped 1 previous similar message [ 6588.937330] Lustre: Skipped 1 previous similar message [ 6593.693747] Lustre: server umount lustre-OST0000 complete [ 6604.997472] Lustre: server umount lustre-MDT0000 complete [ 6608.355751] LustreError: 210776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.404077] LustreError: 210776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 6610.172907] LustreError: 208063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758125 with bad export cookie 4089412313557384659 [ 6610.192320] LustreError: 208063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6610.194677] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.681432] Lustre: server umount lustre-MDT0001 complete [ 6621.069518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.749719] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6639.384498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.519057] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6654.639570] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6661.404269] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6667.613883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6674.209319] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6682.571732] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6687.115377] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6692.912700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 6699.448150] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6699.687783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6702.055580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6702.065972] LustreError: Skipped 1 previous similar message [ 6716.383131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6716.661795] Lustre: server umount lustre-OST0000 complete [ 6723.758620] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6728.030856] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6757.877389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.085276] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6767.561773] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6767.588353] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6767.592084] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6767.606448] Lustre: Skipped 1 previous similar message [ 6771.981609] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6775.764265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6781.714514] Lustre: server umount lustre-OST0000 complete [ 6788.066568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.074699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6788.080256] Lustre: Skipped 5 previous similar messages [ 6788.100315] Lustre: Skipped 11 previous similar messages [ 6795.473043] Lustre: server umount lustre-MDT0000 complete [ 6798.241117] Lustre: 189064:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758297/real 1786758297] req@00000000b59b9451 x1873550974919168/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786758313 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6798.292388] LustreError: 212987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.307836] LustreError: 212987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6799.564449] LustreError: 212995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758314 with bad export cookie 4089412313557385541 [ 6799.569309] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.579723] LustreError: 212995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6800.239729] Lustre: server umount lustre-MDT0001 complete [ 6809.376974] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:45:22 (1786758322) [ 6816.575332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.240110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.254539] Lustre: Skipped 6 previous similar messages [ 6822.351736] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6830.763697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.300232] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6844.940897] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6850.823878] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6858.220220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6866.399748] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6875.092186] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6881.363175] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6881.626678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6887.111624] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6887.238075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6895.987207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6896.279706] Lustre: lustre-OST0001: new disk, initializing [ 6896.285282] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6896.291587] Lustre: 221636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6904.976411] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6906.572851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6906.587978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6906.718281] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6916.096674] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6920.511685] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6920.751447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6925.358661] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6925.590420] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6932.456784] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6932.472263] LustreError: Skipped 1 previous similar message [ 6942.177211] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.369898] Lustre: server umount lustre-OST0001 complete [ 6949.203155] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6949.374823] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6953.855303] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6954.043981] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6982.401617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6987.776494] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6987.798335] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6987.799648] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6988.932814] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 6995.961211] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.434286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7005.623947] Lustre: server umount lustre-OST0001 complete [ 7036.385263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7036.693699] Lustre: server umount lustre-OST0000 complete [ 7044.069637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.071074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.093422] Lustre: Skipped 8 previous similar messages [ 7044.103459] Lustre: Skipped 17 previous similar messages [ 7046.504488] Lustre: server umount lustre-MDT0000 complete [ 7050.106755] LustreError: 218245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758565 with bad export cookie 4089412313557386507 [ 7050.117476] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.122615] LustreError: 218245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7056.498391] Lustre: server umount lustre-MDT0001 complete [ 7078.971201] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:49:52 (1786758592) [ 7084.807641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.191617] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7085.239147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7089.560388] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7096.687490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7096.770409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7096.787862] Lustre: 226882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7100.269804] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7106.980901] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7112.467888] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7119.338095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7119.545750] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7125.585872] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7132.615455] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7141.346520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7141.453546] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7147.671942] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7154.458497] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7155.872038] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7160.506953] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7160.697694] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7165.455472] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7165.628784] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7170.674363] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92f488b0b800.ost_server_uuid 50 [ 7172.344927] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff92f488b0b800.ost_server_uuid in IDLE state after 0 sec [ 7173.348958] Lustre: Permanently deactivating lustre-OST0001 [ 7173.361624] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7176.722780] Lustre: Permanently reactivating lustre-OST0001 [ 7186.075891] Lustre: server umount lustre-OST0001 complete [ 7189.987664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7189.998081] LustreError: Skipped 6 previous similar messages [ 7191.028274] LustreError: 228729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.037673] LustreError: 228729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 7195.567662] Lustre: server umount lustre-OST0000 complete [ 7205.160152] Lustre: server umount lustre-MDT0000 complete [ 7208.795040] LustreError: 226873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758723 with bad export cookie 4089412313557387795 [ 7208.802188] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.806889] LustreError: 226873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7209.281647] Lustre: server umount lustre-MDT0001 complete [ 7232.003678] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:52:25 (1786758745) [ 7238.773260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7238.989526] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7239.039580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7239.059288] Lustre: Skipped 5 previous similar messages [ 7243.995064] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7252.842278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7252.975740] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7253.023923] Lustre: 233575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7257.388544] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7264.883809] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7271.226741] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7278.257128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7278.516668] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7278.551503] Lustre: Permanently deactivating lustre-OST0000 [ 7278.592486] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7285.976976] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7286.291940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7294.215894] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7301.835192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.903395] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7307.989711] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7309.510364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7315.990778] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7317.622523] Lustre: Permanently reactivating lustre-OST0000 [ 7317.634125] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7317.641603] Lustre: Skipped 2 previous similar messages [ 7327.884640] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 00000000b718bd9d, cur 1786758842 deadline 1786758831 last 1786758801 [ 7328.025787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7345.120228] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7345.246090] Lustre: server umount lustre-OST0001 complete [ 7355.074715] Lustre: server umount lustre-OST0000 complete [ 7364.669271] Lustre: server umount lustre-MDT0000 complete [ 7368.336255] LustreError: 233566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758883 with bad export cookie 4089412313557389440 [ 7368.344514] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7368.637395] Lustre: server umount lustre-MDT0001 complete [ 7379.813797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7383.735347] Key type lgssc unregistered [ 7383.998609] LNet: 238459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7384.005572] LNetError: 238459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7384.022955] LNet: Removed LNI 192.168.201.152@tcp [ 7385.284624] Key type .llcrypt unregistered [ 7385.286974] Key type ._llcrypt unregistered [ 7410.564907] Key type ._llcrypt registered [ 7410.568709] Key type .llcrypt registered [ 7410.677945] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7411.929151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7412.107367] alg: No test for adler32 (adler32-zlib) [ 7413.121970] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7413.292461] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7414.927309] Key type lgssc registered [ 7416.463129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7426.211180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7426.250142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.676367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7432.039619] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7437.802686] LustreError: 239924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.818403] LustreError: 239924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7440.652849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.827482] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7440.874977] Lustre: 239914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7440.896836] Lustre: Permanently deactivating lustre-MDT0001 [ 7440.915851] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7441.200385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7443.531634] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7443.541826] LustreError: 239955:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7443.555599] LustreError: 239955:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7443.565915] LustreError: 239955:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7446.134793] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7453.160422] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7457.942605] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7465.236549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7465.650800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7466.663656] LustreError: 241765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7471.173401] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7472.098056] LustreError: 241848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7473.137656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7478.245298] LustreError: 241765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7478.262838] LustreError: 241765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7478.513656] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7483.370840] LustreError: 241848:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.392734] LustreError: 241848:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7485.438507] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7485.710313] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.066516] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7491.094637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7492.005601] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7499.643312] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7503.327350] Lustre: Permanently reactivating lustre-MDT0001 [ 7503.343186] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7503.348551] Lustre: Skipped 2 previous similar messages [ 7505.764866] Lustre: lustre-MDT0001: Client ee79b91e-ae81-4606-8246-e889a913cb8c (at 192.168.201.52@tcp) reconnecting [ 7511.774034] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7513.527655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7519.458795] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7519.709819] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7524.321955] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7524.556206] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7527.064986] Lustre: Permanently deactivating lustre-MDT0001 [ 7527.085410] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7527.090993] Lustre: Skipped 2 previous similar messages [ 7535.785340] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7547.374314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7547.376890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7547.406596] Lustre: Skipped 3 previous similar messages [ 7547.421033] Lustre: Skipped 3 previous similar messages [ 7552.482359] LustreError: 239925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7552.502173] LustreError: 239925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7552.543303] Lustre: server umount lustre-MDT0000 complete [ 7556.459780] LustreError: 239906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759071 with bad export cookie 2785246547082305153 [ 7556.471240] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7556.480338] LustreError: 239906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7557.175480] Lustre: server umount lustre-MDT0001 complete [ 7563.639988] Lustre: server umount lustre-OST0000 complete [ 7575.425462] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:58:08 (1786759088) [ 7585.112936] Lustre: server umount lustre-OST0001 complete [ 7591.066376] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 7600.794538] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7644.871916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.328344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.334367] Lustre: Skipped 2 previous similar messages [ 7645.370178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7645.506657] Lustre: lustre-MDT0000: new disk, initializing [ 7645.597615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7645.621383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7650.153208] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7662.930575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.073069] Lustre: 248230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7663.147159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7663.152444] Lustre: Skipped 1 previous similar message [ 7663.336415] Lustre: lustre-MDT0001: new disk, initializing [ 7663.451135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.518082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7663.544351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7667.341720] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7678.665707] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7686.486885] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7692.918309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7693.111282] Lustre: lustre-OST0000: new disk, initializing [ 7693.113429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7693.116332] Lustre: 250200:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7693.162612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7700.037898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7700.048720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7700.168978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7700.275596] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7710.208284] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7712.647221] LustreError: 248236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7721.446696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.704226] Lustre: lustre-OST0001: new disk, initializing [ 7721.708660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7721.714760] Lustre: 251315:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7722.712367] LustreError: 248236:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7722.759495] LustreError: 248235:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7729.359465] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7729.743211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7729.752603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7729.827653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7732.840367] LustreError: 248235:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7739.100674] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.854561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7745.510428] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.527183] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.528351] LustreError: Skipped 1 previous similar message [ 7745.528991] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7745.588206] Lustre: Skipped 1 previous similar message [ 7750.631921] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7750.654477] Lustre: Skipped 2 previous similar messages [ 7755.747668] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7755.758847] Lustre: Skipped 1 previous similar message [ 7757.279324] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.380614] Lustre: server umount lustre-OST0001 complete [ 7760.873623] LustreError: 250183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7760.900734] LustreError: 250183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7776.261043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7776.269824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7776.284069] Lustre: Skipped 1 previous similar message [ 7776.299202] Lustre: Skipped 1 previous similar message [ 7780.807656] Lustre: server umount lustre-OST0000 complete [ 7786.483652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7786.484898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7786.505475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7786.511542] Lustre: Skipped 1 previous similar message [ 7792.282829] Lustre: server umount lustre-MDT0000 complete [ 7796.704533] LustreError: 248241:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7796.723140] LustreError: 248241:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7796.789205] LustreError: 248222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759311 with bad export cookie 2785246547082307750 [ 7796.796744] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7796.810066] LustreError: 248222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7803.292838] Lustre: server umount lustre-MDT0001 complete [ 7814.961949] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 7818.742174] Key type lgssc unregistered [ 7819.298101] LNet: 253367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7819.320205] LNetError: 253367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7819.347239] LNet: Removed LNI 192.168.201.152@tcp [ 7820.342613] Key type .llcrypt unregistered [ 7820.345734] Key type ._llcrypt unregistered [ 7864.768442] Key type ._llcrypt registered [ 7864.771472] Key type .llcrypt registered [ 7864.861223] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 7865.995960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7866.033423] alg: No test for adler32 (adler32-zlib) [ 7867.337785] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 7867.650462] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 7869.399769] Key type lgssc registered [ 7870.842709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7880.686724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7880.707990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.084514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7882.103824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7882.248148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7887.228262] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7896.213980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7896.279483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7896.302917] Lustre: 255575:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7896.518769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7900.673847] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7907.857550] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7913.822585] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7921.313710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7921.569617] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7921.764945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.991388] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 7931.064369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7937.825607] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.802862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7992.812393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.813181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.837935] Lustre: Skipped 1 previous similar message [ 7995.873202] Lustre: server umount lustre-OST0000 complete [ 7999.713814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8011.180364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8011.474252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.468042] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8012.482137] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8012.485827] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8017.546217] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8025.821052] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8068.474581] Lustre: server umount lustre-OST0000 complete [ 8074.209597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.227331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8074.232519] Lustre: Skipped 2 previous similar messages [ 8079.330240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8079.340389] Lustre: Skipped 1 previous similar message [ 8083.497279] Lustre: server umount lustre-MDT0000 complete [ 8085.472618] Lustre: 254897:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786759584/real 1786759584] req@00000000ed5458d5 x1873553005810944/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786759600 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8085.513132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8085.527743] Lustre: Skipped 1 previous similar message [ 8087.227437] LustreError: 255568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759602 with bad export cookie 13895426533726872936 [ 8087.232698] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.235443] LustreError: 255568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8087.658600] Lustre: server umount lustre-MDT0001 complete [ 8100.460118] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8103.644865] Key type lgssc unregistered [ 8103.984623] LNet: 261003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8103.998846] LNetError: 261003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8104.011586] LNet: Removed LNI 192.168.201.152@tcp [ 8104.758676] Key type .llcrypt unregistered [ 8104.759921] Key type ._llcrypt unregistered [ 8128.248558] Key type ._llcrypt registered [ 8128.251028] Key type .llcrypt registered [ 8128.383967] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8129.470913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8129.495672] alg: No test for adler32 (adler32-zlib) [ 8130.480586] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 8130.708529] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8132.375218] Key type lgssc registered [ 8133.345733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8140.958117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8140.984528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.342295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.857402] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8152.546412] LustreError: 262418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.561051] LustreError: 262418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8157.412680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8157.668091] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8157.814647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8162.634704] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8170.487104] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8176.415422] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8184.888414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8185.390427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.434739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8192.977757] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8201.411839] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8228.326306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.328148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.330796] LustreError: Skipped 1 previous similar message [ 8228.334729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8228.346841] Lustre: Skipped 1 previous similar message [ 8228.404866] Lustre: Skipped 1 previous similar message [ 8232.541684] Lustre: server umount lustre-OST0000 complete [ 8238.568928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8238.581445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8238.597440] Lustre: Skipped 1 previous similar message [ 8238.602301] Lustre: Skipped 1 previous similar message [ 8243.681695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8243.690120] Lustre: Skipped 1 previous similar message [ 8248.800825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.359244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8251.501312] Lustre: server umount lustre-MDT0000 complete [ 8253.923065] LustreError: 262413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8253.932952] LustreError: 262413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8255.281680] LustreError: 262399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759770 with bad export cookie 16613701671438821449 [ 8255.295820] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8255.590293] Lustre: server umount lustre-MDT0001 complete [ 8269.582852] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8272.365555] Key type lgssc unregistered [ 8272.535060] LNet: 266727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8272.540901] LNetError: 266727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8272.550774] LNet: Removed LNI 192.168.201.152@tcp [ 8273.492213] Key type .llcrypt unregistered [ 8273.493846] Key type ._llcrypt unregistered [ 8297.005442] Key type ._llcrypt registered [ 8297.009367] Key type .llcrypt registered [ 8297.142363] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8298.976243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8299.026894] alg: No test for adler32 (adler32-zlib) [ 8300.129453] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 8300.466618] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8302.175252] Key type lgssc registered [ 8304.037842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8313.265910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8313.277972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.511046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.416236] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8324.068870] LustreError: 267952:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8324.080673] LustreError: 267952:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8327.078264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8327.332304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8331.369481] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8339.055533] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8345.798753] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8352.330680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.709572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.792712] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8359.948867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8368.217026] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8382.440396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8382.441423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8382.446115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8382.461823] Lustre: Skipped 1 previous similar message [ 8387.025534] Lustre: server umount lustre-OST0000 complete [ 8392.674105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.688352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8392.699273] Lustre: Skipped 1 previous similar message [ 8397.124720] Lustre: server umount lustre-MDT0000 complete [ 8397.797626] LustreError: 270446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.829549] LustreError: 270446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8401.145837] LustreError: 267932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759916 with bad export cookie 864582847474114769 [ 8401.153128] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8401.161109] LustreError: 267932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8401.483582] Lustre: server umount lustre-MDT0001 complete [ 8414.366743] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8417.090945] Key type lgssc unregistered [ 8417.294796] LNet: 271635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8417.301256] LNetError: 271635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8418.342651] LNet: Removed LNI 192.168.201.152@tcp [ 8419.664537] Key type .llcrypt unregistered [ 8419.668772] Key type ._llcrypt unregistered [ 8451.097314] Key type ._llcrypt registered [ 8451.099518] Key type .llcrypt registered [ 8451.182639] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8452.744635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8452.763073] alg: No test for adler32 (adler32-zlib) [ 8453.888463] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 8454.166278] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8455.815218] Key type lgssc registered [ 8457.281280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8468.016714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8468.027967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.555857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.738574] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8479.721356] LustreError: 273058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.761666] LustreError: 273058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8484.522420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8484.833927] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8484.991450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.184782] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8500.821093] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8506.939851] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8512.927978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8513.173464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8514.230269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8519.234776] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8526.712903] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8555.489850] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8555.496645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.504579] LustreError: Skipped 1 previous similar message [ 8555.507237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8555.529453] Lustre: Skipped 1 previous similar message [ 8555.559050] Lustre: Skipped 1 previous similar message [ 8558.073523] Lustre: server umount lustre-OST0000 complete [ 8565.733150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8565.754214] Lustre: Skipped 1 previous similar message [ 8565.755574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8565.770904] Lustre: Skipped 1 previous similar message [ 8568.112613] Lustre: server umount lustre-MDT0000 complete [ 8570.849849] LustreError: 273057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8570.880915] LustreError: 273057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8571.849916] LustreError: 273037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760086 with bad export cookie 11545658760365815145 [ 8571.857298] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8571.862735] LustreError: 273037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8572.184857] Lustre: server umount lustre-MDT0001 complete [ 8584.097946] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8586.680979] Key type lgssc unregistered [ 8587.001919] LNet: 277368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8587.015633] LNetError: 277368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8588.091522] LNet: Removed LNI 192.168.201.152@tcp [ 8588.934135] Key type .llcrypt unregistered [ 8588.935127] Key type ._llcrypt unregistered [ 8609.359896] Key type ._llcrypt registered [ 8609.363896] Key type .llcrypt registered [ 8609.504279] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8611.333739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8611.368687] alg: No test for adler32 (adler32-zlib) [ 8612.393791] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 8612.601802] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8614.383191] Key type lgssc registered [ 8615.348430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8624.170877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8624.188810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.728850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.397771] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8635.873287] LustreError: 278607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.881894] LustreError: 278607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8638.138625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8638.572948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8643.557858] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8652.386781] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8658.272172] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8665.229641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8665.759994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.914685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8674.602973] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8684.712813] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8698.854632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.857875] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8698.878297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8698.887822] Lustre: Skipped 1 previous similar message [ 8703.272539] Lustre: server umount lustre-OST0000 complete [ 8709.092688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8709.094809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8709.121955] Lustre: Skipped 2 previous similar messages [ 8709.131104] Lustre: Skipped 1 previous similar message [ 8714.209703] LustreError: 278611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8714.237421] LustreError: 278611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8715.562742] Lustre: server umount lustre-MDT0000 complete [ 8718.310404] LustreError: 278589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8718.341851] LustreError: 278589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8719.481628] LustreError: 279366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760234 with bad export cookie 7358153016685654660 [ 8719.484809] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8719.493326] LustreError: 279366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8719.994878] Lustre: server umount lustre-MDT0001 complete [ 8729.608206] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 8732.309436] Key type lgssc unregistered [ 8732.604688] LNet: 282312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8732.622890] LNetError: 282312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8732.645189] LNet: Removed LNI 192.168.201.152@tcp [ 8733.410272] Key type .llcrypt unregistered [ 8733.412408] Key type ._llcrypt unregistered [ 8742.740711] Key type ._llcrypt registered [ 8742.742898] Key type .llcrypt registered [ 8742.885970] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8757.575300] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 8773.288758] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 8774.505740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8774.518983] alg: No test for adler32 (adler32-zlib) [ 8775.529268] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 8775.753115] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 8777.441605] Key type lgssc registered [ 8778.795700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8815.941388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8815.983610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.249955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8817.290217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8817.387833] Lustre: lustre-MDT0000: new disk, initializing [ 8817.444455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8817.469685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8821.289940] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8830.505524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.639252] Lustre: 285866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8830.702541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8830.707383] Lustre: Skipped 1 previous similar message [ 8830.798848] Lustre: lustre-MDT0001: new disk, initializing [ 8830.900603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8830.930915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8830.939219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8834.631288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8843.352197] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8848.242811] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8854.394758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.575982] Lustre: lustre-OST0000: new disk, initializing [ 8854.579690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8854.583065] Lustre: 287830:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8854.641213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.764802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8861.777810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8861.916794] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8862.038810] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8873.301924] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8877.495398] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8877.729935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8882.694994] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8882.901610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8887.776771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8887.780085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.782937] LustreError: Skipped 1 previous similar message [ 8887.785374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.835458] Lustre: Skipped 1 previous similar message [ 8891.000465] Lustre: server umount lustre-OST0000 complete [ 8897.504541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8897.509771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.541176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8897.545366] Lustre: Skipped 1 previous similar message [ 8900.970216] Lustre: server umount lustre-MDT0000 complete [ 8904.709313] LustreError: 285858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760419 with bad export cookie 6761653185465829400 [ 8904.715492] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8904.726629] LustreError: 285858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8905.208234] Lustre: server umount lustre-MDT0001 complete [ 8914.240746] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:20:27 (1786760427) [ 8921.495200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8921.920603] LustreError: 290244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.942896] LustreError: 290244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8922.020328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8927.200943] LustreError: 290245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8928.270581] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8931.313773] LustreError: 290244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.418082] LustreError: 290245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.994128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.299222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8941.919841] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8948.874565] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8953.320551] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8959.194936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8959.420499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8963.811427] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 8970.816727] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8981.988424] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8981.992021] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.993435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8981.993441] Lustre: Skipped 2 previous similar messages [ 8982.055328] Lustre: Skipped 2 previous similar messages [ 8986.823332] Lustre: server umount lustre-OST0000 complete [ 8992.225469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.229858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8992.241033] Lustre: Skipped 1 previous similar message [ 8992.255903] Lustre: Skipped 1 previous similar message [ 8997.339511] Lustre: server umount lustre-MDT0000 complete [ 8997.354972] LustreError: 290972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8997.380172] LustreError: 290972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9001.320194] LustreError: 293028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760516 with bad export cookie 6761653185465830282 [ 9001.325885] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9001.336310] LustreError: 293028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9001.899698] Lustre: server umount lustre-MDT0001 complete [ 9012.010364] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9014.086591] Key type lgssc unregistered [ 9014.327924] LNet: 293783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9014.333035] LNetError: 293783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9015.398537] LNet: Removed LNI 192.168.201.152@tcp [ 9016.125507] Key type .llcrypt unregistered [ 9016.127797] Key type ._llcrypt unregistered [ 9042.209634] Key type ._llcrypt registered [ 9042.211560] Key type .llcrypt registered [ 9042.303402] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9056.131271] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9057.228036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9057.408443] alg: No test for adler32 (adler32-zlib) [ 9058.527638] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 9058.824661] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9060.559316] Key type lgssc registered [ 9061.671615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9116.409145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9116.441653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.952867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9118.004435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9118.143273] Lustre: lustre-MDT0000: new disk, initializing [ 9118.319760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9118.361769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9124.783879] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9137.089954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.303515] Lustre: 297594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9137.412877] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9137.420310] Lustre: Skipped 1 previous similar message [ 9137.500436] Lustre: lustre-MDT0001: new disk, initializing [ 9137.597908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9137.648910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9137.655625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9142.914350] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9153.311452] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9159.422682] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9165.209066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9165.457283] Lustre: lustre-OST2710: new disk, initializing [ 9165.462150] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9165.466361] Lustre: 299564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9165.529244] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9170.981993] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9175.090487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9175.101673] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9175.233348] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9180.658677] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9187.453123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9187.675546] Lustre: lustre-OST03e8: new disk, initializing [ 9187.683704] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9187.690323] Lustre: 300630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9187.790511] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9195.134521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9195.149707] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9195.320141] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9195.324347] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9203.160585] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9210.857703] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9211.138515] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9216.126940] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9216.290628] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9221.927623] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9222.180242] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9227.227360] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9227.462269] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9231.338263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.355189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9231.356226] Lustre: Skipped 1 previous similar message [ 9231.379533] Lustre: Skipped 3 previous similar messages [ 9233.540654] Lustre: server umount lustre-MDT0000 complete [ 9236.735076] LustreError: 300628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760751 with bad export cookie 5135350930859765743 [ 9236.742331] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9236.745089] LustreError: 300628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9237.037158] Lustre: server umount lustre-MDT0001 complete [ 9250.968916] Lustre: server umount lustre-OST2710 complete [ 9265.810695] Lustre: server umount lustre-OST03e8 complete [ 9284.483259] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [ 9296.071636] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9336.747244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9336.998953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9337.066943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9337.155537] Lustre: lustre-MDT0000: new disk, initializing [ 9337.243664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.259888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9343.137960] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9355.961120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9356.185637] Lustre: 305881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9356.261649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9356.268684] Lustre: Skipped 1 previous similar message [ 9356.381624] Lustre: lustre-MDT0001: new disk, initializing [ 9356.536751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9356.570680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9356.591948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9362.708121] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9375.621498] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9382.857233] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9391.337278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.673365] Lustre: lustre-OST0000: new disk, initializing [ 9391.676989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9391.682905] Lustre: 307850:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9391.800572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.315411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9397.325723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9397.530895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9397.865343] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9410.049619] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9415.830232] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9416.022139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9420.489718] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9420.728776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9423.328563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9423.331656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.334589] LustreError: Skipped 1 previous similar message [ 9423.348348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.358963] Lustre: Skipped 3 previous similar messages [ 9428.451451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9428.470750] Lustre: Skipped 2 previous similar messages [ 9433.571976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9433.584303] Lustre: Skipped 1 previous similar message [ 9437.152800] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9437.424815] Lustre: server umount lustre-OST0000 complete [ 9443.808860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9443.811669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9443.820588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9443.829658] Lustre: Skipped 1 previous similar message [ 9447.587660] Lustre: server umount lustre-MDT0000 complete [ 9448.944874] LustreError: 307046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.970063] LustreError: 307046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9451.640301] LustreError: 305873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760966 with bad export cookie 5135350930859767423 [ 9451.670908] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9452.232923] Lustre: server umount lustre-MDT0001 complete [ 9462.909416] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:29:35 (1786760975) [ 9464.389789] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9466.007774] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:29:39 (1786760979) [ 9470.542185] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9500.370751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.040551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9501.081507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9501.298861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.915187] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9516.874295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.040273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9517.088247] Lustre: 311709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9520.885616] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9543.980083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.154639] Lustre: Denying initial registration attempt from nid 192.168.201.152@tcp, specified as failover [ 9544.182319] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9544.210785] LustreError: 313165:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9544.228294] LustreError: 313165:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9544.235916] LustreError: 313165:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9544.359229] Lustre: server umount lustre-OST0000 complete [ 9544.361102] LustreError: 313165:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9553.387652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9553.405515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.459039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.461777] Lustre: Skipped 1 previous similar message [ 9559.520345] LustreError: 311718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.576629] LustreError: 311718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9559.933561] Lustre: server umount lustre-MDT0000 complete [ 9564.640774] LustreError: 313385:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.690786] LustreError: 313385:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9566.543411] LustreError: 311702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761081 with bad export cookie 5135350930859768305 [ 9566.566554] LustreError: 311702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9566.594054] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9567.337477] Lustre: server umount lustre-MDT0001 complete [ 9582.780868] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9586.067827] Key type lgssc unregistered [ 9586.433769] LNet: 314289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9586.460461] LNetError: 314289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9586.490400] LNet: Removed LNI 192.168.201.152@tcp [ 9587.781084] Key type .llcrypt unregistered [ 9587.812707] Key type ._llcrypt unregistered [ 9603.695327] Key type ._llcrypt registered [ 9603.697322] Key type .llcrypt registered [ 9603.864063] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9605.022722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9605.042885] alg: No test for adler32 (adler32-zlib) [ 9606.026247] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 9606.327717] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9608.145790] Key type lgssc registered [ 9609.745751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9660.901975] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9670.406092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9670.414902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.740533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9671.783483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9672.029421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9677.537885] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9688.090247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.195561] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9688.222993] Lustre: 317055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9688.533635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9693.796443] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9702.163895] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9708.446840] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9728.557203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.711862] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9728.844601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.991395] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9744.052772] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9746.914108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9746.917951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.918506] LustreError: Skipped 1 previous similar message [ 9746.925695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9746.936120] Lustre: Skipped 1 previous similar message [ 9752.040421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9752.052104] Lustre: Skipped 2 previous similar messages [ 9752.345629] Lustre: server umount lustre-OST0000 complete [ 9757.153103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.171640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9757.175974] Lustre: Skipped 1 previous similar message [ 9762.272124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9762.933667] Lustre: server umount lustre-MDT0000 complete [ 9767.727416] LustreError: 317047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761282 with bad export cookie 12091709736304239671 [ 9767.738128] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9767.750780] LustreError: 317047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9768.098690] Lustre: server umount lustre-MDT0001 complete [ 9786.275722] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [ 9789.874746] Key type lgssc unregistered [ 9790.262301] LNet: 320649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9790.273479] LNetError: 320649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9790.297569] LNet: Removed LNI 192.168.201.152@tcp [ 9791.371247] Key type .llcrypt unregistered [ 9791.376466] Key type ._llcrypt unregistered [ 9819.919679] Key type ._llcrypt registered [ 9819.927119] Key type .llcrypt registered [ 9820.067984] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [ 9821.301719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9821.346027] alg: No test for adler32 (adler32-zlib) [ 9822.516677] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [ 9822.753718] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [ 9824.455663] Key type lgssc registered [ 9825.280379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9835.576664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9835.627792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9837.535276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.545049] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9847.800717] LustreError: 322065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9847.818602] LustreError: 322065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9852.608672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9852.989459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9858.322542] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9868.106490] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9875.067803] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9885.046187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.469961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9890.895673] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9898.226380] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9919.457067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9919.467559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.491410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9920.690518] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 9921.508857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.529873] Lustre: Skipped 1 previous similar message [ 9925.768078] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 9925.776666] Lustre: Skipped 3 previous similar messages [ 9930.906515] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [ 9930.909523] Lustre: Skipped 3 previous similar messages [ 9932.255268] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9932.471349] Lustre: server umount lustre-MDT0000 complete [ 9936.053229] LustreError: 322060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9936.992033] LustreError: 322047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761451 with bad export cookie 3271915773536659023 [ 9936.992702] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.006551] LustreError: 322047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9941.134075] LustreError: 322061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9941.143140] LustreError: 322061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9941.192757] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [ 9941.196157] Lustre: Skipped 3 previous similar messages [ 9941.992209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.807888] Lustre: server umount lustre-MDT0001 complete [ 9952.171606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9961.926557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.167325] LustreError: 325668:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9963.179886] LustreError: 325668:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000c695a55f x1873555055750144/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786761478 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9963.209641] LustreError: 325668:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9963.774628] LustreError: 323901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9963.787748] LustreError: 323901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9963.862625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9963.909866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9967.846235] LustreError: 323902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.968860] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9972.969369] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9978.338299] LustreError: 323901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9978.364116] LustreError: 323901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9979.043764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.450831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9982.538794] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9984.581486] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [ 9994.168974] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9995.778699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10001.877325] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10003.759853] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10010.080446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10010.083347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.088916] LustreError: Skipped 1 previous similar message [10010.090710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10010.090717] Lustre: Skipped 1 previous similar message [10010.117581] Lustre: Skipped 1 previous similar message [10023.391195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10023.764786] Lustre: server umount lustre-OST0000 complete [10028.517299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10028.541742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10028.564884] Lustre: Skipped 8 previous similar messages [10034.607931] Lustre: server umount lustre-MDT0000 complete [10038.717616] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761553 with bad export cookie 3271915773536666548 [10038.728649] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10038.734239] LustreError: 322046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10039.431713] Lustre: server umount lustre-MDT0001 complete [10052.587433] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10055.725216] Key type lgssc unregistered [10056.049771] LNet: 328363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10056.066965] LNetError: 328363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10056.086804] LNet: Removed LNI 192.168.201.152@tcp [10056.786587] Key type .llcrypt unregistered [10056.788633] Key type ._llcrypt unregistered [10087.848712] Key type ._llcrypt registered [10087.851927] Key type .llcrypt registered [10088.052741] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10089.504392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10089.596136] alg: No test for adler32 (adler32-zlib) [10090.774659] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [10091.156090] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10092.879320] Key type lgssc registered [10094.361831] Lustre: Echo OBD driver; http://www.lustre.org/ [10103.066658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10103.087056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10104.688623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10109.985966] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10115.040338] LustreError: 329783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10115.048747] LustreError: 329783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10119.627562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.046459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10125.967865] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10135.070550] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10141.147532] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10145.766800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10145.773702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.814081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10149.866421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.867939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10154.977530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10154.989461] Lustre: Skipped 2 previous similar messages [10160.095174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10160.096915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10160.112057] Lustre: Skipped 1 previous similar message [10160.312758] Lustre: server umount lustre-MDT0000 complete [10164.193810] LustreError: 330933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.206621] LustreError: 330933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10164.248950] LustreError: 329763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761679 with bad export cookie 3472025662830015798 [10164.270278] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10165.024644] Lustre: server umount lustre-MDT0001 complete [10180.396338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.645685] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10180.671640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10180.918636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10186.358428] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10196.201315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.496632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10196.578534] Lustre: 332370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10196.857279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10200.994463] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10208.488160] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10214.477167] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10224.048121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10224.369774] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10224.379047] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10224.385339] LustreError: 334253:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10224.390040] LustreError: 334253:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10224.394062] LustreError: 334253:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10224.529483] Lustre: server umount lustre-OST0000 complete [10224.539080] LustreError: 334253:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10236.104529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10236.286820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10236.462369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10242.162028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10242.803862] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10250.586289] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10259.353249] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10259.496542] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10259.614575] Lustre: lustre-OST0001: new disk, initializing [10259.617562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10259.622854] Lustre: 335797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10259.803042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10266.272200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10266.295544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10266.474324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10266.677589] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10274.905471] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10281.954290] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10281.960333] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10281.988186] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10281.996039] Lustre: Skipped 1 previous similar message [10284.819349] Lustre: server umount lustre-OST0001 complete [10287.085246] LustreError: 334722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10287.116177] LustreError: 334722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10292.192581] LustreError: 334721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10292.193062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10292.209214] LustreError: 334721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10292.218215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10292.220816] Lustre: Skipped 1 previous similar message [10292.247735] Lustre: Skipped 2 previous similar messages [10296.593176] Lustre: server umount lustre-OST0000 complete [10302.437020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.461375] Lustre: Skipped 1 previous similar message [10307.326105] Lustre: server umount lustre-MDT0000 complete [10307.554995] LustreError: 332379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10307.583269] LustreError: 332379:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10311.980309] LustreError: 332364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761826 with bad export cookie 3472025662830016351 [10311.984449] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10311.992724] LustreError: 332364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10312.505605] Lustre: server umount lustre-MDT0001 complete [10325.409394] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10328.494679] Key type lgssc unregistered [10328.746690] LNet: 337762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10328.757791] LNetError: 337762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10328.780136] LNet: Removed LNI 192.168.201.152@tcp [10329.456094] Key type .llcrypt unregistered [10329.458723] Key type ._llcrypt unregistered [10409.168620] Key type ._llcrypt registered [10409.177493] Key type .llcrypt registered [10409.397358] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [10426.931706] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10428.572358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10428.594811] alg: No test for adler32 (adler32-zlib) [10429.707386] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [10430.012925] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10431.687211] Key type lgssc registered [10432.929193] Lustre: Echo OBD driver; http://www.lustre.org/ [10471.875595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10471.893664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.146703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10473.185703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10473.321207] Lustre: lustre-MDT0000: new disk, initializing [10473.452972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10473.489669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10478.244461] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10490.272840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.395996] Lustre: 342701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10490.470861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10490.480458] Lustre: Skipped 1 previous similar message [10490.567297] Lustre: lustre-MDT0001: new disk, initializing [10490.658434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10490.701714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10490.717704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10495.940514] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10506.645383] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10513.055200] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10521.061188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10521.352614] Lustre: lustre-OST0000: new disk, initializing [10521.357744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10521.375605] Lustre: 344667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10521.515095] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10527.704736] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10527.795489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10527.809396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10527.980039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10538.619238] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10544.696771] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10544.995448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10550.759589] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10551.097725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10553.832418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10553.838159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.840782] LustreError: Skipped 1 previous similar message [10553.842509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10553.886887] Lustre: Skipped 1 previous similar message [10558.962638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10558.973029] Lustre: Skipped 2 previous similar messages [10559.737613] Lustre: server umount lustre-OST0000 complete [10567.670991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10567.681549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10567.697513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.186510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10570.565667] Lustre: server umount lustre-MDT0000 complete [10575.320264] LustreError: 342692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762090 with bad export cookie 10621543166565211376 [10575.323511] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10575.337068] LustreError: 342692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10575.939309] Lustre: server umount lustre-MDT0001 complete [10585.719317] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:48:18 (1786762098) [10595.786621] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:48:28 (1786762108) [10605.721417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10606.103220] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10606.110797] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10606.163889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10609.862986] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10611.169989] LustreError: 347427:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.289752] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10618.088000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.611923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10622.963061] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10631.878074] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10637.803950] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10646.038328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10646.696065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10654.185822] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10663.875971] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10673.033732] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10675.043898] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10676.989923] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10679.009117] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10681.405982] Lustre: Failing over lustre-MDT0000 [10681.516909] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [10681.524435] Lustre: Skipped 2 previous similar messages [10683.720383] Lustre: server umount lustre-MDT0000 complete [10683.873749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.897503] LustreError: 347422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10683.912447] LustreError: 347422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10688.995758] LustreError: 347421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.013930] LustreError: 347421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10699.233243] Lustre: 340927:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786762198/real 1786762198] req@00000000b07b7474 x1873555692266496/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786762214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10699.242776] LustreError: 347423:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10699.269410] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10699.305748] LustreError: 347423:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10704.390258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.479507] LustreError: 340924:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000447f2e0 x1873555692271872/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10709.984278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10710.060287] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10713.250659] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10715.115053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10715.151026] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10715.184460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10715.709836] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10725.739601] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.653603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10731.604254] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10734.624527] Lustre: Failing over lustre-MDT0000 [10734.963897] Lustre: server umount lustre-MDT0000 complete [10735.586281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10735.587797] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.602625] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.614810] Lustre: Skipped 4 previous similar messages [10735.652496] LustreError: 347426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10751.969062] Lustre: 340926:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786762250/real 1786762250] req@0000000062c0897a x1873555692287104/t0(0) o400->MGC192.168.201.152@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786762266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10751.991795] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10756.341043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.188892] LustreError: 340924:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004205b634 x1873555692291968/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10761.727619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10761.846588] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10764.445483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10766.600329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10766.890240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10766.897295] Lustre: Skipped 2 previous similar messages [10766.950318] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10766.998944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10777.261231] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10779.307704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10787.302348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10787.305767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10787.330550] Lustre: Skipped 2 previous similar messages [10787.336989] Lustre: Skipped 1 previous similar message [10792.145560] Lustre: server umount lustre-MDT0000 complete [10792.464399] LustreError: 347422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.499622] LustreError: 347422:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [10797.029890] LustreError: 347408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762311 with bad export cookie 10621543166565214085 [10797.044337] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10797.050361] LustreError: 347408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10797.407710] Lustre: server umount lustre-MDT0001 complete [10811.977214] Lustre: server umount lustre-OST0000 complete [10830.666686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10831.239318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10836.916776] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10846.525328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10852.214863] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10860.943059] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10868.201929] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10875.478168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10876.852947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10883.791524] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [10893.806402] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10901.477768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.494250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10901.501890] Lustre: Skipped 1 previous similar message [10901.520138] Lustre: Skipped 2 previous similar messages [10907.802451] Lustre: server umount lustre-OST0000 complete [10913.761712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.187907] Lustre: server umount lustre-MDT0000 complete [10921.952392] LustreError: 353768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.978024] LustreError: 353768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10923.018395] LustreError: 353755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762437 with bad export cookie 10621543166565214834 [10923.023816] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.032255] LustreError: 353755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10923.367957] Lustre: server umount lustre-MDT0001 complete [10935.307340] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [10939.293319] Key type lgssc unregistered [10939.785519] LNet: 357310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10939.789865] LNetError: 357310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10940.839896] LNet: Removed LNI 192.168.201.152@tcp [10941.996250] Key type .llcrypt unregistered [10941.997908] Key type ._llcrypt unregistered [10974.131807] Key type ._llcrypt registered [10974.133537] Key type .llcrypt registered [10974.231701] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [10975.567251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10975.693341] alg: No test for adler32 (adler32-zlib) [10976.958928] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [10977.348672] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [10979.079456] Key type lgssc registered [10980.713897] Lustre: Echo OBD driver; http://www.lustre.org/ [10989.627688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10989.648520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.172836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10996.184923] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11001.319318] LustreError: 358827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.346195] LustreError: 358827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11005.474384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11005.853829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11010.707432] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11020.197993] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11027.105114] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11035.902197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11036.447042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11037.515231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11044.608417] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11055.090899] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11067.365381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11067.381044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11067.403702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11069.011414] Lustre: server umount lustre-MDT0000 complete [11071.464933] LustreError: 358827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.468149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.376647] LustreError: 358808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762589 with bad export cookie 5403286881831106892 [11074.390681] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11074.393316] LustreError: 358808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11075.117530] Lustre: server umount lustre-MDT0001 complete [11090.840259] Lustre: server umount lustre-OST0000 complete [11107.615766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11108.127936] LustreError: 362547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.144647] LustreError: 362547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11108.213642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11113.447234] LustreError: 362548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11113.798512] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11118.581438] LustreError: 362547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11125.515290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11126.437769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11134.238647] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11144.061562] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11150.352577] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11158.674559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11159.140037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11164.918288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11172.274803] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11178.621820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11197.920913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11197.928776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.950726] Lustre: Skipped 1 previous similar message [11197.959533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11199.462156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.101276] Lustre: server umount lustre-MDT0000 complete [11204.578404] LustreError: 362547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.598789] LustreError: 362547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11205.918814] LustreError: 362529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762720 with bad export cookie 5403286881831108005 [11205.931457] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11205.935487] LustreError: 362529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11206.474892] Lustre: server umount lustre-MDT0001 complete [11221.358836] Lustre: server umount lustre-OST0000 complete [11239.079474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11239.474723] LustreError: 366483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11239.552884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11246.295640] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11257.008959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11262.335501] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11270.031304] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11276.571354] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11283.612195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.846226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11283.853355] Lustre: Skipped 1 previous similar message [11284.932937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11290.190373] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11298.289921] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11305.455889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11305.459916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.471805] Lustre: Skipped 2 previous similar messages [11305.479775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11305.487647] Lustre: Skipped 2 previous similar messages [11307.489114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11309.621743] Lustre: server umount lustre-OST0000 complete [11317.736107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.744845] Lustre: Skipped 1 previous similar message [11317.752237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.765627] Lustre: Skipped 3 previous similar messages [11320.602859] Lustre: server umount lustre-MDT0000 complete [11322.848543] LustreError: 366484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.874296] LustreError: 366484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11324.412983] LustreError: 366464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762839 with bad export cookie 5403286881831109027 [11324.416047] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11324.424727] LustreError: 366464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11324.673615] Lustre: server umount lustre-MDT0001 complete [11335.737194] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11338.543922] Key type lgssc unregistered [11338.896686] LNet: 370017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11338.900826] LNetError: 370017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11338.918073] LNet: Removed LNI 192.168.201.152@tcp [11339.679745] Key type .llcrypt unregistered [11339.681673] Key type ._llcrypt unregistered [11369.101470] Key type ._llcrypt registered [11369.105961] Key type .llcrypt registered [11369.209707] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11370.695119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11370.748956] alg: No test for adler32 (adler32-zlib) [11371.830828] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [11372.113120] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11373.807803] Key type lgssc registered [11374.938214] Lustre: Echo OBD driver; http://www.lustre.org/ [11382.685689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11382.727267] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11382.738192] LustreError: 371470:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11382.774821] LustreError: 371470:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11382.784609] LustreError: 371470:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11382.804071] LustreError: 371470:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11382.817345] LustreError: 371470:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11390.953573] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11390.958207] LustreError: 371868:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11390.981388] LustreError: 371868:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11390.991242] LustreError: 371868:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11391.009290] LustreError: 371868:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11391.024615] LustreError: 371868:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11411.447215] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11414.526950] Key type lgssc unregistered [11414.971693] LNet: 372874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11414.978697] LNetError: 372874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11414.994660] LNet: Removed LNI 192.168.201.152@tcp [11415.841184] Key type .llcrypt unregistered [11415.843119] Key type ._llcrypt unregistered [11438.097793] Key type ._llcrypt registered [11438.100397] Key type .llcrypt registered [11438.213354] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [11454.058464] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11455.069204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11455.102041] alg: No test for adler32 (adler32-zlib) [11456.174729] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [11456.388371] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11458.039274] Key type lgssc registered [11459.604420] Lustre: Echo OBD driver; http://www.lustre.org/ [11499.546336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11499.589805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11500.815955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11500.876358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11500.957766] Lustre: lustre-MDT0000: new disk, initializing [11501.020764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11501.033578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11505.197778] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11518.184670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.338946] Lustre: 376037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11518.425847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11518.437839] Lustre: Skipped 1 previous similar message [11518.504882] Lustre: lustre-MDT0001: new disk, initializing [11518.647886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11518.706777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11518.722315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11524.040477] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11535.973497] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11542.227074] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11549.241394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.538730] Lustre: lustre-OST0000: new disk, initializing [11549.544627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11549.559458] Lustre: 378004:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11549.705495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11555.981576] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11556.939495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11556.953652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11557.089608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11564.890496] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11569.332288] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11569.513312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11574.471580] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11574.663660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11577.825804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11577.826396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.835303] LustreError: Skipped 1 previous similar message [11577.839313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11577.873684] Lustre: Skipped 1 previous similar message [11582.754992] Lustre: server umount lustre-OST0000 complete [11588.064704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.083122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.090313] Lustre: Skipped 2 previous similar messages [11592.823105] Lustre: server umount lustre-MDT0000 complete [11597.190821] LustreError: 376028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763112 with bad export cookie 14690872436898631229 [11597.198263] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11597.201585] LustreError: 376028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11597.642831] Lustre: server umount lustre-MDT0001 complete [11607.517625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11609.505124] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:05:22 (1786763122) [11617.083521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11617.725014] LustreError: 380462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11617.754965] LustreError: 380462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11617.871439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11622.889181] LustreError: 380463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.697532] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11628.001864] LustreError: 380462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11632.033828] LustreError: 380463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.829310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11636.470048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11642.684415] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11652.622083] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11660.032883] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11668.279666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11668.748731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11675.282749] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11684.957958] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11693.021977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11693.236167] Lustre: lustre-OST0001: new disk, initializing [11693.239740] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11693.244784] Lustre: 383300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11693.346268] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11698.829306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11698.850197] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11699.046981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11700.140586] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11709.622962] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11714.545320] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11714.547151] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11714.557114] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11714.567044] Lustre: Skipped 2 previous similar messages [11714.569511] LustreError: Skipped 1 previous similar message [11719.467483] Lustre: server umount lustre-OST0001 complete [11719.661961] LustreError: 382284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11719.676600] LustreError: 382284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11729.909358] LustreError: 383308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11729.931176] LustreError: 383308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11737.569449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11737.579388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11737.590745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11737.595692] Lustre: Skipped 2 previous similar messages [11743.813389] Lustre: server umount lustre-OST0000 complete [11749.346700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11749.363294] LustreError: Skipped 1 previous similar message [11749.376672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.388651] Lustre: Skipped 1 previous similar message [11749.400816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11749.403350] Lustre: Skipped 3 previous similar messages [11754.069780] Lustre: server umount lustre-MDT0000 complete [11755.489513] LustreError: 380462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.526441] LustreError: 380462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11758.027960] LustreError: 380443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763272 with bad export cookie 14690872436898632118 [11758.031650] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11758.038468] LustreError: 380443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11772.383128] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11772.714712] Lustre: server umount lustre-MDT0001 complete [11783.304840] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [11787.019666] Key type lgssc unregistered [11787.323664] LNet: 385293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11787.330633] LNetError: 385293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11787.345119] LNet: Removed LNI 192.168.201.152@tcp [11788.064871] Key type .llcrypt unregistered [11788.066556] Key type ._llcrypt unregistered [11842.732373] Key type ._llcrypt registered [11842.738676] Key type .llcrypt registered [11842.844633] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [11844.084437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11844.112362] alg: No test for adler32 (adler32-zlib) [11845.320711] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [11845.575578] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [11847.263155] Key type lgssc registered [11848.617610] Lustre: Echo OBD driver; http://www.lustre.org/ [11857.627811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11857.654174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.851589] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11858.867900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11858.979748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11863.333329] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11871.487841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11871.610057] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11871.645971] Lustre: 388095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11871.873816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11877.089824] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11884.790283] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11890.330251] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11892.195189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11892.204174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11892.227627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11895.266164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.267424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11895.294510] Lustre: Skipped 1 previous similar message [11898.331197] Lustre: server umount lustre-MDT0000 complete [11901.986096] LustreError: 388090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763416 with bad export cookie 15774011098562802654 [11901.995192] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11902.001366] LustreError: 388090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11902.356530] Lustre: server umount lustre-MDT0001 complete [11911.654229] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:10:24 (1786763424) [11918.305660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.885633] LustreError: 390610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.899886] LustreError: 390610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11918.941394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11923.257040] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11923.942467] LustreError: 390611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.067051] LustreError: 390611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.222266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11931.594103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11936.262674] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11943.397241] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11948.804108] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11957.352541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.826248] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11958.044754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11964.443370] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [11971.956952] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11981.002852] Lustre: Permanently deactivating lustre-OST0000 [11981.040317] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11991.084228] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11993.832565] LustreError: 393693:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11993.837187] LustreError: 393693:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11993.840752] LustreError: 393693:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11996.164906] LustreError: 393742:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11996.168803] LustreError: 393742:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12004.093739] Lustre: server umount lustre-OST0000 complete [12006.858852] LustreError: 393991:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12006.870594] LustreError: 393991:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12008.416606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12008.420531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12008.440089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.036714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12013.037400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12018.144698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12018.154880] Lustre: Skipped 2 previous similar messages [12019.254940] Lustre: server umount lustre-MDT0000 complete [12022.876159] LustreError: 390590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763537 with bad export cookie 15774011098562803235 [12022.887377] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12022.889487] LustreError: 390590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12032.791918] Lustre: server umount lustre-MDT0001 complete [12041.772490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12046.907449] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12050.542702] LustreError: 395150:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12057.450617] Lustre: 395489:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.152@tcp [12059.741147] Lustre: server umount MGS complete [12070.014350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12070.550172] LustreError: 396215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.573038] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12070.578564] Lustre: Skipped 1 previous similar message [12070.646790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12075.220297] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12076.010134] LustreError: 396216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.620915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.829864] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12087.275463] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12095.178449] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12101.474837] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12103.430385] Lustre: Permanently reactivating lustre-OST0000 [12103.448936] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12103.456892] Lustre: Skipped 2 previous similar messages [12131.809229] LustreError: 398000:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12139.199268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12139.511174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12139.518477] Lustre: Skipped 1 previous similar message [12145.477862] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12154.066203] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12161.994752] Lustre: server umount lustre-OST0000 complete [12166.113406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12166.118354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12166.142320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12175.549446] Lustre: server umount lustre-MDT0000 complete [12177.375605] Lustre: 387419:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786763676/real 1786763676] req@00000000862d8220 x1873557177074432/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786763692 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12177.379831] LustreError: 396211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.405712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.440194] LustreError: 396211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12177.470389] Lustre: Skipped 1 previous similar message [12179.349398] LustreError: 396195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763694 with bad export cookie 15774011098562804285 [12179.358460] LustreError: 396195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12179.360673] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12179.730859] Lustre: server umount lustre-MDT0001 complete [12190.768505] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12193.485549] Key type lgssc unregistered [12193.757858] LNet: 400630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12193.765157] LNetError: 400630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12193.779229] LNet: Removed LNI 192.168.201.152@tcp [12194.497329] Key type .llcrypt unregistered [12194.499227] Key type ._llcrypt unregistered [12210.387501] Key type ._llcrypt registered [12210.390400] Key type .llcrypt registered [12210.506453] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [12225.314921] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12227.031923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12227.078909] alg: No test for adler32 (adler32-zlib) [12228.107360] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [12228.300573] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12229.983242] Key type lgssc registered [12230.932743] Lustre: Echo OBD driver; http://www.lustre.org/ [12264.683047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12264.718175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12266.007240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12266.059410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12266.139995] Lustre: lustre-MDT0000: new disk, initializing [12266.240647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12266.303288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12271.903770] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12282.771409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12282.876353] Lustre: 403793:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12282.922844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12282.925584] Lustre: Skipped 1 previous similar message [12282.990804] Lustre: lustre-MDT0001: new disk, initializing [12283.133852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12283.212272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12283.225852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12287.672197] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12296.378834] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12302.009758] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12308.003455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12308.232433] Lustre: lustre-OST0000: new disk, initializing [12308.236046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12308.242677] Lustre: 405761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12308.346302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12314.251809] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12316.211568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12316.219117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12316.260366] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12324.297352] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12328.574271] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12328.774230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12333.380572] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12333.585500] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12336.608993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12336.609871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12336.623567] LustreError: Skipped 1 previous similar message [12336.624544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12336.631582] Lustre: Skipped 1 previous similar message [12336.658711] Lustre: Skipped 1 previous similar message [12341.548365] Lustre: server umount lustre-OST0000 complete [12346.849261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12346.851338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12346.864842] Lustre: Skipped 1 previous similar message [12351.165555] Lustre: server umount lustre-MDT0000 complete [12355.126952] LustreError: 403787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763870 with bad export cookie 14543141170249080011 [12355.136515] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12355.537393] Lustre: server umount lustre-MDT0001 complete [12365.617736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12368.173591] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:18:00 (1786763880) [12375.358678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12375.689981] LustreError: 408223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12375.701062] LustreError: 408223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12375.742973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12381.153558] LustreError: 408224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.309731] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12385.256250] LustreError: 408223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.372049] LustreError: 408224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12390.457583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12390.973964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12396.641456] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12406.951760] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12413.948268] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12420.892119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12421.334486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.857857] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12437.292585] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12440.134774] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12447.145643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12451.300159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12451.306775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12451.322038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12451.332833] Lustre: Skipped 1 previous similar message [12451.349190] Lustre: Skipped 2 previous similar messages [12456.416405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12456.430196] Lustre: Skipped 1 previous similar message [12457.605271] Lustre: server umount lustre-OST0000 complete [12461.539060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12461.547379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12461.561028] Lustre: Skipped 1 previous similar message [12467.684804] Lustre: server umount lustre-MDT0000 complete [12471.615829] LustreError: 408204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786763986 with bad export cookie 14543141170249080886 [12471.622205] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12471.629228] LustreError: 408204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12471.930656] Lustre: server umount lustre-MDT0001 complete [12482.894797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12486.153202] Key type lgssc unregistered [12486.426615] LNet: 411954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12486.432488] LNetError: 411954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12486.453787] LNet: Removed LNI 192.168.201.152@tcp [12487.329359] Key type .llcrypt unregistered [12487.335523] Key type ._llcrypt unregistered [12516.294892] Key type ._llcrypt registered [12516.301027] Key type .llcrypt registered [12516.506665] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12516.972310] Key type .llcrypt unregistered [12516.975745] Key type ._llcrypt unregistered [12536.338231] Key type ._llcrypt registered [12536.342366] Key type .llcrypt registered [12536.463442] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12537.536712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12537.556375] alg: No test for adler32 (adler32-zlib) [12538.757855] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [12539.212556] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12540.977392] Key type lgssc registered [12542.557730] Lustre: Echo OBD driver; http://www.lustre.org/ [12551.163838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12551.185276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.736324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12558.233834] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12562.915828] LustreError: 414171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12562.938962] LustreError: 414171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12566.823415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12567.118343] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12567.135954] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12567.147904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12572.643398] LustreError: 414168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12572.672623] LustreError: 414168:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12574.603883] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12577.761863] LustreError: 414171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12577.785257] LustreError: 414171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12577.785333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12582.884241] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12582.913417] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12583.720750] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12591.374440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12591.832593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12596.256696] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12605.613739] Lustre: server umount lustre-OST0000 complete [12612.577265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12612.578537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12612.580132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12612.596272] Lustre: Skipped 1 previous similar message [12617.701526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12617.704132] Lustre: Skipped 1 previous similar message [12619.930669] Lustre: server umount lustre-MDT0000 complete [12622.831498] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.863027] LustreError: 414172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12623.846051] Lustre: 413483:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786764122/real 1786764122] req@0000000090dba2e4 x1873557904713728/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786764138 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12623.880938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.156695] LustreError: 416278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764139 with bad export cookie 2296881780523190169 [12624.165205] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12624.174571] LustreError: 416278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12624.657299] Lustre: server umount lustre-MDT0001 complete [12637.450166] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12639.996782] Key type lgssc unregistered [12640.312356] LNet: 417320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12640.326645] LNetError: 417320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12640.336732] LNet: Removed LNI 192.168.201.152@tcp [12641.201095] Key type .llcrypt unregistered [12641.204117] Key type ._llcrypt unregistered [12668.300975] Key type ._llcrypt registered [12668.305359] Key type .llcrypt registered [12668.422649] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12669.828535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12669.893963] alg: No test for adler32 (adler32-zlib) [12671.010886] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [12671.209642] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12672.879630] Key type lgssc registered [12673.789323] Lustre: Echo OBD driver; http://www.lustre.org/ [12683.940632] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12683.995933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12699.683208] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12704.799527] LustreError: 418714:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.152@tcp: failed processing log, type 4: rc = -110 [12731.552394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12737.696093] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12745.688870] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12752.602534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.714631] LustreError: 418050:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001e93adb2 x1873558043101440/t0(0) o250->MGC192.168.201.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12753.184856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12753.264510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12757.336965] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12758.498951] LustreError: 418740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12758.526860] LustreError: 418740:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12763.618975] LustreError: 418739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.976208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12766.408297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12770.950887] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12778.583599] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12784.219560] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12791.777220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12791.785846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.801126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12793.330821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12793.331447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12793.370570] Lustre: Skipped 1 previous similar message [12796.918215] Lustre: server umount lustre-OST0000 complete [12802.016513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12802.026032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12802.045330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12807.227816] Lustre: server umount lustre-MDT0000 complete [12808.672340] LustreError: 419755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12808.693941] LustreError: 419755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12810.857681] LustreError: 418720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764325 with bad export cookie 250901491200996527 [12810.863553] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12810.866412] LustreError: 418720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12811.103859] Lustre: server umount lustre-MDT0001 complete [12823.756847] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12826.889842] Key type lgssc unregistered [12827.165896] LNet: 422287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12827.171184] LNetError: 422287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12827.186605] LNet: Removed LNI 192.168.201.152@tcp [12827.985550] Key type .llcrypt unregistered [12827.991516] Key type ._llcrypt unregistered [12859.794967] Key type ._llcrypt registered [12859.799609] Key type .llcrypt registered [12859.895993] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12860.406168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12860.437528] alg: No test for adler32 (adler32-zlib) [12861.618777] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [12861.861717] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12863.559228] Key type lgssc registered [12864.621809] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.811858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.841793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.341297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.271452] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12883.936989] LustreError: 423707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.951851] LustreError: 423707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12887.192376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.760630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12891.879419] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12899.553290] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12904.692145] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12910.892262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12911.146670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12917.366082] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [12918.320738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12924.523704] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12928.480158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12928.484176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12928.496862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12928.500538] Lustre: Skipped 2 previous similar messages [12931.239799] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.52@tcp (stopping) [12933.607774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12933.613575] Lustre: Skipped 2 previous similar messages [12934.174164] Lustre: server umount lustre-MDT0000 complete [12936.129046] LustreError: 426284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.52@tcp arrived at 1786764451 with bad export cookie 16719839979019169566 [12936.137212] LustreError: 426284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12937.557881] LustreError: 423686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786764452 with bad export cookie 16719839979019168670 [12937.565601] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12937.566303] LustreError: 423686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12937.698104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12937.706911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.720436] Lustre: Skipped 2 previous similar messages [12937.729161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12938.721208] LustreError: 424434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12938.744611] LustreError: 424434:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12942.753769] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.763308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12942.769886] LustreError: 423706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12942.798290] Lustre: Skipped 1 previous similar message [12943.874868] Lustre: server umount lustre-OST0000 complete [12947.873857] LustreError: 423703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12947.886216] LustreError: 423703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12950.536226] Lustre: server umount lustre-MDT0001 complete [12960.214526] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [12962.905914] Key type lgssc unregistered [12963.271865] LNet: 427391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12963.288628] LNetError: 427391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12964.330598] LNet: Removed LNI 192.168.201.152@tcp [12965.159934] Key type .llcrypt unregistered [12965.161997] Key type ._llcrypt unregistered [12987.457443] Key type ._llcrypt registered [12987.462332] Key type .llcrypt registered [12987.590335] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [12988.713563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12988.791106] alg: No test for adler32 (adler32-zlib) [12989.872318] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [12990.106238] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [12991.769531] Key type lgssc registered [12993.073880] Lustre: Echo OBD driver; http://www.lustre.org/ [13000.329120] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13000.345125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13001.668050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13005.888530] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13011.949340] LustreError: 428790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.975145] LustreError: 428790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13014.450087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13014.876677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13020.258533] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13028.219607] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13034.413131] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13040.755737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13041.113944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13046.261580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13047.279978] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13054.947758] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13061.097099] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13061.103756] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13061.115460] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13062.796026] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13064.673836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13064.687260] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.934617] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13067.957347] Lustre: Skipped 1 previous similar message [13073.062050] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13073.069406] Lustre: Skipped 2 previous similar messages [13074.399606] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13074.697416] Lustre: server umount lustre-MDT0001 complete [13074.920718] LustreError: 430628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13074.941613] LustreError: 430628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13080.034034] LustreError: 430890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13084.799268] Lustre: server umount lustre-OST0000 complete [13085.156345] LustreError: 428790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.170733] LustreError: 428790:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13099.167606] Lustre: server umount lustre-MDT0000 complete [13111.471846] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13113.896389] Key type lgssc unregistered [13114.152955] LNet: 432480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13114.160923] LNetError: 432480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13114.171928] LNet: Removed LNI 192.168.201.152@tcp [13114.929342] Key type .llcrypt unregistered [13114.931163] Key type ._llcrypt unregistered [13138.328919] Key type ._llcrypt registered [13138.333763] Key type .llcrypt registered [13138.442358] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [13152.080512] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13153.609429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13153.685248] alg: No test for adler32 (adler32-zlib) [13154.736486] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [13154.919922] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13156.576115] Key type lgssc registered [13157.683433] Lustre: Echo OBD driver; http://www.lustre.org/ [13198.639528] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13215.258784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13215.312804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13216.672685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13216.719616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13216.835566] Lustre: lustre-MDT0000: new disk, initializing [13216.938462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13216.965240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13222.350689] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13238.720795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13239.119706] Lustre: lustre-OST0000: new disk, initializing [13239.123061] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13239.126807] Lustre: Skipped 1 previous similar message [13239.132138] Lustre: 437496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13239.210595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13246.006151] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13248.094804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13248.105153] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13248.135065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13255.902500] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13271.968768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13272.079732] Lustre: 436475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13272.129437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13272.234676] Lustre: lustre-MDT0001: new disk, initializing [13272.401547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13272.448205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13272.461589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13276.210027] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13285.060954] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13285.353118] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13289.675230] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13290.000867] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13296.772481] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.52@tcp (stopping) [13299.686973] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13299.689470] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13299.742741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13299.755484] Lustre: Skipped 1 previous similar message [13301.516963] Lustre: server umount lustre-MDT0001 complete [13307.017634] LustreError: 436485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.041352] LustreError: 436485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13309.924127] LustreError: 436488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13309.943495] LustreError: 436488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13315.041112] LustreError: 436485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13315.073917] LustreError: 436485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13320.171967] LustreError: 436483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13324.768662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13324.775319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13324.787612] Lustre: Skipped 1 previous similar message [13324.798488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13325.283085] LustreError: 438833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13325.294208] LustreError: 438833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13327.203411] Lustre: server umount lustre-OST0000 complete [13337.887325] Lustre: server umount lustre-MDT0000 complete [13352.604163] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13355.820449] Key type lgssc unregistered [13356.065973] LNet: 441249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13356.073679] LNetError: 441249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13356.093531] LNet: Removed LNI 192.168.201.152@tcp [13356.869246] Key type .llcrypt unregistered [13356.872207] Key type ._llcrypt unregistered [13401.575276] Key type ._llcrypt registered [13401.577453] Key type .llcrypt registered [13401.663883] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [13401.976555] Key type .llcrypt unregistered [13401.978339] Key type ._llcrypt unregistered [13417.252371] Key type ._llcrypt registered [13417.256188] Key type .llcrypt registered [13417.361335] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [13418.325450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13418.391851] alg: No test for adler32 (adler32-zlib) [13419.463098] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [13419.667900] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [13421.351938] Key type lgssc registered [13422.708890] Lustre: Echo OBD driver; http://www.lustre.org/ [13453.417411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13453.459529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.858810] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13454.883286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13454.992673] Lustre: lustre-MDT0000: new disk, initializing [13455.125188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13455.147173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13460.510114] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13471.115095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13471.309500] Lustre: 444631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13471.379098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13471.383698] Lustre: Skipped 1 previous similar message [13471.478583] Lustre: lustre-MDT0001: new disk, initializing [13471.621969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13471.670894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13471.686900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13476.381628] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13486.174544] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13492.272966] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13501.560178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13501.968782] Lustre: lustre-OST0000: new disk, initializing [13501.984296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13502.003789] Lustre: 446603:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13502.128300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13510.323603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13510.353793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13510.572516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13511.394816] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13523.164357] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13531.108648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13531.111245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13531.133191] Lustre: Skipped 1 previous similar message [13531.155886] Lustre: Skipped 2 previous similar messages [13536.234661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13536.245317] Lustre: Skipped 2 previous similar messages [13536.651184] Lustre: server umount lustre-MDT0000 complete [13541.173838] LustreError: 446601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765056 with bad export cookie 14608095328010603398 [13541.196692] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13541.346118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13541.359462] Lustre: Skipped 1 previous similar message [13541.364966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13541.540815] Lustre: server umount lustre-MDT0001 complete [13547.657044] Lustre: server umount lustre-OST0000 complete [13560.565813] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:37:53 (1786765073) [13569.872381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13570.367971] LustreError: 448630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13570.388346] LustreError: 448630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13570.468849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13575.150415] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13575.650964] LustreError: 448631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13580.771507] LustreError: 448630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.405055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.887552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13589.933488] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13597.609079] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13604.817437] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13626.096213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13626.319451] Lustre: Found index 0 for lustre-OST0000, updating log [13626.534834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13633.537432] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [13641.904671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13642.303152] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13650.403847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13650.420057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13650.430136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13655.688516] Lustre: server umount lustre-OST0000 complete [13671.906171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13671.921217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13671.938490] Lustre: Skipped 1 previous similar message [13671.944720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13671.947969] Lustre: Skipped 3 previous similar messages [13677.944602] Lustre: server umount lustre-MDT0000 complete [13678.049079] LustreError: 448631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13678.066369] LustreError: 448631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13681.461490] LustreError: 448612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765196 with bad export cookie 14608095328010604784 [13681.467587] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13681.477614] LustreError: 448612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [13681.875498] Lustre: server umount lustre-MDT0001 complete [13690.474473] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:40:03 (1786765203) [13696.166645] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [14312.153558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14312.476467] LustreError: 455047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14312.497899] LustreError: 455047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14312.545444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14317.585755] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14318.059534] LustreError: 455048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14322.148602] LustreError: 455047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14327.611640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14328.082085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14332.541955] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14341.104197] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14347.700227] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14369.404815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14369.822949] Lustre: Found index 0 for lustre-OST0000, updating log [14370.168652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14373.089305] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [14376.319784] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14384.393817] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14387.502494] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14387.527168] Lustre: Skipped 1 previous similar message [14387.541911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14387.545160] Lustre: Skipped 2 previous similar messages [14392.857785] Lustre: server umount lustre-OST0000 complete [14397.860900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14397.866059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14397.881313] Lustre: Skipped 1 previous similar message [14397.890085] Lustre: Skipped 1 previous similar message [14399.981085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14399.991414] Lustre: Skipped 3 previous similar messages [14404.128702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14404.132756] Lustre: Skipped 6 previous similar messages [14411.231690] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14411.407931] Lustre: server umount lustre-MDT0000 complete [14412.329348] LustreError: 455048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14412.351278] LustreError: 455048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14415.408865] LustreError: 455027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786765930 with bad export cookie 14608095328010605876 [14415.421814] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14415.427865] LustreError: 455027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14415.777347] Lustre: server umount lustre-MDT0001 complete [14433.506080] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp [14447.102907] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [14457.177184] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [14493.024173] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:53:26 (1786766006) [14510.358687] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [14513.420771] Key type lgssc unregistered [14513.669750] LNet: 461663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14513.674431] LNetError: 461663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14632.496695] LNet: Removed LNI 192.168.201.152@tcp [14633.519856] Key type .llcrypt unregistered [14633.521631] Key type ._llcrypt unregistered [14651.400642] Key type ._llcrypt registered [14651.402251] Key type .llcrypt registered [14651.535629] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [14652.987211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14653.018478] alg: No test for adler32 (adler32-zlib) [14654.059345] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [14654.320805] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [14656.018052] Key type lgssc registered [14657.571705] Lustre: Echo OBD driver; http://www.lustre.org/ [14662.047630] LNet: Added LNI 192.168.201.152@tcp100 [8/256/0/180] [14663.034073] LNet: Added LNI 192.168.201.152@tcp101 [8/256/0/180] [14663.935070] LNet: Added LNI 192.168.201.152@tcp102 [8/256/0/180] [14665.015213] LNet: Added LNI 192.168.201.152@tcp103 [8/256/0/180] [14666.034737] LNet: Added LNI 192.168.201.152@tcp104 [8/256/0/180] [14667.010434] LNet: Added LNI 192.168.201.152@tcp105 [8/256/0/180] [14667.968378] LNet: Added LNI 192.168.201.152@tcp106 [8/256/0/180] [14668.904532] LNet: Added LNI 192.168.201.152@tcp107 [8/256/0/180] [14669.728708] LNet: Added LNI 192.168.201.152@tcp108 [8/256/0/180] [14670.688736] LNet: Added LNI 192.168.201.152@tcp109 [8/256/0/180] [14671.674736] LNet: Added LNI 192.168.201.152@tcp110 [8/256/0/180] [14672.393549] LNet: Added LNI 192.168.201.152@tcp111 [8/256/0/180] [14673.214977] LNet: Added LNI 192.168.201.152@tcp112 [8/256/0/180] [14674.130954] LNet: Added LNI 192.168.201.152@tcp113 [8/256/0/180] [14675.130162] LNet: Added LNI 192.168.201.152@tcp114 [8/256/0/180] [14676.093307] LNet: Added LNI 192.168.201.152@tcp115 [8/256/0/180] [14677.022678] LNet: Added LNI 192.168.201.152@tcp116 [8/256/0/180] [14677.933761] LNet: Added LNI 192.168.201.152@tcp117 [8/256/0/180] [14678.773244] LNet: Added LNI 192.168.201.152@tcp118 [8/256/0/180] [14679.761701] LNet: Added LNI 192.168.201.152@tcp119 [8/256/0/180] [14680.575728] LNet: Added LNI 192.168.201.152@tcp120 [8/256/0/180] [14681.574945] LNet: Added LNI 192.168.201.152@tcp121 [8/256/0/180] [14682.357116] LNet: Added LNI 192.168.201.152@tcp122 [8/256/0/180] [14683.434601] LNet: Added LNI 192.168.201.152@tcp123 [8/256/0/180] [14684.132441] LNet: Added LNI 192.168.201.152@tcp124 [8/256/0/180] [14684.831788] LNet: Added LNI 192.168.201.152@tcp125 [8/256/0/180] [14685.830122] LNet: Added LNI 192.168.201.152@tcp126 [8/256/0/180] [14686.725855] LNet: Added LNI 192.168.201.152@tcp127 [8/256/0/180] [14687.645159] LNet: Added LNI 192.168.201.152@tcp128 [8/256/0/180] [14688.805828] LNet: Added LNI 192.168.201.152@tcp129 [8/256/0/180] [14689.677676] LNet: Added LNI 192.168.201.152@tcp130 [8/256/0/180] [14690.488955] LNet: Added LNI 192.168.201.152@tcp131 [8/256/0/180] [14691.459850] LNet: Added LNI 192.168.201.152@tcp132 [8/256/0/180] [14692.497436] LNet: Added LNI 192.168.201.152@tcp133 [8/256/0/180] [14693.675313] LNet: Added LNI 192.168.201.152@tcp134 [8/256/0/180] [14694.640503] LNet: Added LNI 192.168.201.152@tcp135 [8/256/0/180] [14701.810757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14701.833809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14702.953204] Lustre: 464696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [14703.059111] LNetError: 464717:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14703.087149] LNetError: Protocol error connecting to 192.168.201.152@tcp124 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14703.099248] LNetError: Skipped 4 previous similar messages [14703.188433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14703.246664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14703.314342] Lustre: lustre-MDT0000: new disk, initializing [14703.442541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14703.500389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14703.526912] LNet: 4 local NIs in recovery (showing 4): 192.168.201.152@tcp119, 192.168.201.152@tcp127, 192.168.201.152@tcp129, 192.168.201.152@tcp128 [14704.160716] LNetError: 464726:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14704.168945] LNetError: 464726:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14704.175620] LNetError: Protocol error connecting to 192.168.201.152@tcp118 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14704.189541] LNetError: Skipped 2 previous similar messages [14706.598874] LNetError: 462155:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14706.616209] LNetError: 462155:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [14706.629089] LNetError: Protocol error connecting to 192.168.201.152@tcp119 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14706.642320] LNetError: Skipped 5 previous similar messages [14708.744866] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14710.689132] LNetError: 464717:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14710.708932] LNetError: 464717:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14710.718360] LNetError: Protocol error connecting to 192.168.201.152@tcp119 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14718.880535] LNetError: 464721:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14718.888423] LNetError: Protocol error connecting to 192.168.201.152@tcp119 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14719.496961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14719.685373] Lustre: 464714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14719.763104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14719.768563] Lustre: Skipped 1 previous similar message [14719.870485] Lustre: lustre-MDT0001: new disk, initializing [14719.929772] Lustre: 465516:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [14719.942790] Lustre: 465516:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14720.129703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14720.189733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14720.208779] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14724.476448] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14727.723750] Lustre: 465900:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [14727.736770] Lustre: 465900:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14733.152432] LNetError: 464716:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14733.161139] LNetError: Protocol error connecting to 192.168.201.152@tcp120 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14734.635918] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14739.900777] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14746.035126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14746.284411] Lustre: lustre-OST0000: new disk, initializing [14746.288966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14746.294856] Lustre: 466688:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14746.364819] Lustre: 466693:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [14746.368569] Lustre: 466693:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14746.403249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14749.536889] LNetError: 464716:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14749.540196] LNetError: 464716:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [14749.547926] LNetError: Protocol error connecting to 192.168.201.152@tcp118 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14749.558412] LNetError: Skipped 9 previous similar messages [14752.616645] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [14754.326616] Lustre: 467144:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.152@tcp: can't add NID 192.168.201.152@tcp130: rc = -75 [14754.333820] Lustre: 467144:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14754.382733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14754.392348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14754.511673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14761.420338] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14763.935178] LNet: 1 local NIs in recovery (showing 1): 192.168.201.152@tcp119 [14770.150383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14770.168087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14770.183419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14775.287878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14775.297146] Lustre: Skipped 2 previous similar messages [14775.682405] Lustre: server umount lustre-OST0000 complete [14780.387889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14780.412042] Lustre: Skipped 1 previous similar message [14780.417381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14780.426990] Lustre: Skipped 1 previous similar message [14784.355575] LNetError: 464721:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.152 [14784.371390] LNetError: 464721:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 14 previous similar messages [14784.380412] LNetError: Protocol error connecting to 192.168.201.152@tcp120 at host 192.168.201.152:988: is it running a compatible version of Lustre? [14784.388197] LNetError: Skipped 14 previous similar messages [14785.504911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14785.520264] Lustre: Skipped 1 previous similar message [14786.026254] Lustre: server umount lustre-MDT0000 complete [14790.272578] LustreError: 465525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786766305 with bad export cookie 2186071224337261278 [14790.281870] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14790.291773] LustreError: 465525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14790.751812] Lustre: server umount lustre-MDT0001 complete [14802.280756] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [14805.132389] Key type lgssc unregistered [14805.582639] LNet: 468605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14805.591705] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14805.600331] LNet: Removed LNI 192.168.201.152@tcp [14806.623246] LNet: Removed LNI 192.168.201.152@tcp100 [14806.625358] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14806.635832] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14806.645072] LNet: Removed LNI 192.168.201.152@tcp101 [14807.711674] LNet: Removed LNI 192.168.201.152@tcp102 [14807.718058] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14807.726351] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14807.731301] LNet: Removed LNI 192.168.201.152@tcp103 [14808.735212] LNet: Removed LNI 192.168.201.152@tcp104 [14809.759843] LNet: Removed LNI 192.168.201.152@tcp105 [14809.770499] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14809.785398] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14809.802598] LNet: Removed LNI 192.168.201.152@tcp106 [14810.847162] LNet: Removed LNI 192.168.201.152@tcp107 [14811.871206] LNet: Removed LNI 192.168.201.152@tcp108 [14812.895920] LNet: Removed LNI 192.168.201.152@tcp109 [14813.919212] LNet: Removed LNI 192.168.201.152@tcp110 [14813.924033] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14813.934035] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14813.939730] LNet: Removed LNI 192.168.201.152@tcp111 [14814.943494] LNet: Removed LNI 192.168.201.152@tcp112 [14815.967185] LNet: Removed LNI 192.168.201.152@tcp113 [14817.001869] LNet: Removed LNI 192.168.201.152@tcp114 [14818.015222] LNet: Removed LNI 192.168.201.152@tcp115 [14819.039259] LNet: Removed LNI 192.168.201.152@tcp116 [14820.063472] LNet: Removed LNI 192.168.201.152@tcp117 [14821.093104] LNet: Removed LNI 192.168.201.152@tcp118 [14821.097462] LNet: Removed LNI 192.168.201.152@tcp119 [14822.111185] LNet: Removed LNI 192.168.201.152@tcp120 [14822.115039] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14822.121445] LNetError: 468605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [14822.128951] LNet: Removed LNI 192.168.201.152@tcp121 [14823.135330] LNet: Removed LNI 192.168.201.152@tcp122 [14824.159573] LNet: Removed LNI 192.168.201.152@tcp123 [14825.184663] LNet: Removed LNI 192.168.201.152@tcp124 [14826.207220] LNet: Removed LNI 192.168.201.152@tcp125 [14827.231185] LNet: Removed LNI 192.168.201.152@tcp126 [14828.256487] LNet: Removed LNI 192.168.201.152@tcp127 [14829.279157] LNet: Removed LNI 192.168.201.152@tcp128 [14830.303756] LNet: Removed LNI 192.168.201.152@tcp129 [14830.310507] LNet: Removed LNI 192.168.201.152@tcp130 [14830.314645] LNet: Removed LNI 192.168.201.152@tcp131 [14830.322587] LNet: Removed LNI 192.168.201.152@tcp132 [14830.326958] LNet: Removed LNI 192.168.201.152@tcp133 [14830.331643] LNet: Removed LNI 192.168.201.152@tcp134 [14830.379347] LNet: Removed LNI 192.168.201.152@tcp135 [14831.194593] Key type .llcrypt unregistered [14831.198816] Key type ._llcrypt unregistered [14860.008926] Key type ._llcrypt registered [14860.011990] Key type .llcrypt registered [14860.144765] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [14876.910898] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [14878.467884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14878.523459] alg: No test for adler32 (adler32-zlib) [14879.615565] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [14879.953968] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [14881.623200] Key type lgssc registered [14882.812783] Lustre: Echo OBD driver; http://www.lustre.org/ [14922.707517] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:00:35 (1786766435) [14942.898674] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [14945.677272] Key type lgssc unregistered [14946.021154] LNet: 472699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14946.028935] LNetError: 472699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14946.061265] LNet: Removed LNI 192.168.201.152@tcp [14947.400131] Key type .llcrypt unregistered [14947.403688] Key type ._llcrypt unregistered [14969.829129] Key type ._llcrypt registered [14969.831415] Key type .llcrypt registered [14969.956434] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [14971.876112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14971.925150] alg: No test for adler32 (adler32-zlib) [14973.054459] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [14973.333890] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [14975.119526] Key type lgssc registered [14976.298944] Lustre: Echo OBD driver; http://www.lustre.org/ [14986.857996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14986.920674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14988.251517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14988.301439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14988.424378] Lustre: lustre-MDT0000: new disk, initializing [14988.529870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14988.544217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14993.745654] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15004.751233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15004.975931] Lustre: 473924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15005.103341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15005.117035] Lustre: Skipped 1 previous similar message [15005.238428] Lustre: lustre-MDT0001: new disk, initializing [15005.344863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15005.391837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15005.413519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15010.884899] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15022.416234] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15028.234647] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15035.280898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15035.670837] Lustre: lustre-OST0000: new disk, initializing [15035.677679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15035.688564] Lustre: 475897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15035.805752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15042.146369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15042.154911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15042.322509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15043.037732] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15053.739333] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15058.725856] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing lnet_if_list [15064.954891] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15064.996554] LNet: Added LNI 192.168.201.152@tcp10 [8/256/0/180] [15070.774849] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15070.817281] LNet: Added LNI 192.168.201.152@tcp11 [8/256/0/180] [15076.320340] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15076.351887] LNet: Added LNI 192.168.201.152@tcp12 [8/256/0/180] [15081.009943] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15081.044972] LNet: Added LNI 192.168.201.152@tcp13 [8/256/0/180] [15085.657220] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15085.680692] LNet: Added LNI 192.168.201.152@tcp14 [8/256/0/180] [15090.620759] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15090.662072] LNet: Added LNI 192.168.201.152@tcp15 [8/256/0/180] [15095.956824] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15095.991923] LNet: Added LNI 192.168.201.152@tcp16 [8/256/0/180] [15100.937437] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15100.956191] LNet: Added LNI 192.168.201.152@tcp17 [8/256/0/180] [15105.495735] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15105.535446] LNet: Added LNI 192.168.201.152@tcp18 [8/256/0/180] [15111.399413] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15111.454113] LNet: Added LNI 192.168.201.152@tcp19 [8/256/0/180] [15117.151828] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15117.199219] LNet: Added LNI 192.168.201.152@tcp20 [8/256/0/180] [15122.640989] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15122.684220] LNet: Added LNI 192.168.201.152@tcp21 [8/256/0/180] [15128.920111] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15128.982325] LNet: Added LNI 192.168.201.152@tcp22 [8/256/0/180] [15135.498666] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15135.524443] LNet: Added LNI 192.168.201.152@tcp23 [8/256/0/180] [15144.669529] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15144.723145] LNet: Added LNI 192.168.201.152@tcp24 [8/256/0/180] [15154.495613] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15154.606970] LNet: Added LNI 192.168.201.152@tcp25 [8/256/0/180] [15162.024250] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15162.045677] LNet: Added LNI 192.168.201.152@tcp26 [8/256/0/180] [15168.184839] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15168.206739] LNet: Added LNI 192.168.201.152@tcp27 [8/256/0/180] [15174.035671] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15174.055522] LNet: Added LNI 192.168.201.152@tcp28 [8/256/0/180] [15181.231347] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15181.278179] LNet: Added LNI 192.168.201.152@tcp29 [8/256/0/180] [15188.750421] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15188.802726] LNet: Added LNI 192.168.201.152@tcp30 [8/256/0/180] [15196.465627] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15196.500746] LNet: Added LNI 192.168.201.152@tcp31 [8/256/0/180] [15203.265889] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15203.314106] LNet: Added LNI 192.168.201.152@tcp32 [8/256/0/180] [15209.456273] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15209.475364] LNet: Added LNI 192.168.201.152@tcp33 [8/256/0/180] [15216.250837] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15216.288269] LNet: Added LNI 192.168.201.152@tcp34 [8/256/0/180] [15222.918800] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15222.963369] LNet: Added LNI 192.168.201.152@tcp35 [8/256/0/180] [15229.522572] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15229.552046] LNet: Added LNI 192.168.201.152@tcp36 [8/256/0/180] [15236.182813] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15236.209083] LNet: Added LNI 192.168.201.152@tcp37 [8/256/0/180] [15242.943551] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15242.990806] LNet: Added LNI 192.168.201.152@tcp38 [8/256/0/180] [15251.777670] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15251.818291] LNet: Added LNI 192.168.201.152@tcp39 [8/256/0/180] [15259.713912] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15259.741079] LNet: Added LNI 192.168.201.152@tcp40 [8/256/0/180] [15266.630652] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15266.657394] LNet: Added LNI 192.168.201.152@tcp41 [8/256/0/180] [15275.712614] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15275.811052] LNet: Added LNI 192.168.201.152@tcp42 [8/256/0/180] [15284.868255] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15284.898398] LNet: Added LNI 192.168.201.152@tcp43 [8/256/0/180] [15294.037770] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15294.058237] LNet: Added LNI 192.168.201.152@tcp44 [8/256/0/180] [15304.397495] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15304.442697] LNet: Added LNI 192.168.201.152@tcp45 [8/256/0/180] [15314.990935] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15315.014530] LNet: Added LNI 192.168.201.152@tcp46 [8/256/0/180] [15324.684371] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15324.725425] LNet: Added LNI 192.168.201.152@tcp47 [8/256/0/180] [15335.929745] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15335.955755] LNet: Added LNI 192.168.201.152@tcp48 [8/256/0/180] [15345.134179] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15345.157275] LNet: Added LNI 192.168.201.152@tcp49 [8/256/0/180] [15355.092496] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15355.139988] LNet: Added LNI 192.168.201.152@tcp50 [8/256/0/180] [15374.304683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15374.310638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15374.331345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15375.333152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15375.334562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15375.357284] Lustre: Skipped 2 previous similar messages [15380.548880] Lustre: server umount lustre-MDT0000 complete [15384.334575] LustreError: 473918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786766899 with bad export cookie 12227573060546735849 [15384.346559] LustreError: 473918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15384.348657] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15384.708708] Lustre: server umount lustre-MDT0001 complete [15401.951256] Lustre: 473198:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786766900/real 1786766900] req@000000000a376f58 x1873560457099520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786766916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15402.018563] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15402.026410] Lustre: Skipped 1 previous similar message [15406.047531] Lustre: 473200:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786766905/real 1786766905] req@00000000d225e2a2 x1873560457099776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786766921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15410.143085] Lustre: 473198:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786766909/real 1786766909] req@0000000012088bff x1873560457100032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786766925 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15410.165651] Lustre: 473198:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15410.505686] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15415.328143] Lustre: 473198:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786766914/real 1786766914] req@00000000d434b4b3 x1873560457100160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786766930 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15422.632853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15430.112398] LustreError: 475889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15430.141412] LustreError: 475889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15430.260380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15431.269716] LustreError: 475890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15431.274255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15436.306674] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15436.772291] LustreError: 475889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15441.897151] LustreError: 475890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15447.014309] LustreError: 475889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15447.041764] LustreError: 475889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15447.811374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15448.220983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15453.692670] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15455.109683] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15464.302873] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15472.339707] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15494.113612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15494.128994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15494.188374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15499.815334] Lustre: server umount lustre-OST0000 complete [15504.352697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15504.368196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15504.384792] Lustre: Skipped 1 previous similar message [15504.397129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15504.412336] Lustre: Skipped 1 previous similar message [15509.993477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15509.996304] Lustre: Skipped 2 previous similar messages [15510.371307] Lustre: server umount lustre-MDT0000 complete [15514.083674] LustreError: 500143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15514.113458] LustreError: 500143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15514.463540] LustreError: 473917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786767029 with bad export cookie 12227573060546738397 [15514.464279] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15514.470944] LustreError: 473917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15514.773653] Lustre: server umount lustre-MDT0001 complete [15526.580982] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [15529.747589] Key type lgssc unregistered [15530.230842] LNet: 502945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15530.235928] LNetError: 502945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15530.240525] LNet: Removed LNI 192.168.201.152@tcp [15530.242922] LNet: Removed LNI 192.168.201.152@tcp10 [15530.245037] LNet: Removed LNI 192.168.201.152@tcp11 [15530.247239] LNet: Removed LNI 192.168.201.152@tcp12 [15530.250255] LNet: Removed LNI 192.168.201.152@tcp13 [15530.252973] LNet: Removed LNI 192.168.201.152@tcp14 [15530.256742] LNet: Removed LNI 192.168.201.152@tcp15 [15530.261722] LNet: Removed LNI 192.168.201.152@tcp16 [15530.281556] LNet: Removed LNI 192.168.201.152@tcp17 [15530.291535] LNet: Removed LNI 192.168.201.152@tcp18 [15530.297850] LNet: Removed LNI 192.168.201.152@tcp19 [15530.305772] LNet: Removed LNI 192.168.201.152@tcp20 [15530.310816] LNet: Removed LNI 192.168.201.152@tcp21 [15530.319453] LNet: Removed LNI 192.168.201.152@tcp22 [15530.326616] LNet: Removed LNI 192.168.201.152@tcp23 [15530.330897] LNet: Removed LNI 192.168.201.152@tcp24 [15530.334612] LNet: Removed LNI 192.168.201.152@tcp25 [15530.343722] LNet: Removed LNI 192.168.201.152@tcp26 [15530.346723] LNet: Removed LNI 192.168.201.152@tcp27 [15530.348565] LNet: Removed LNI 192.168.201.152@tcp28 [15530.351423] LNet: Removed LNI 192.168.201.152@tcp29 [15530.353265] LNet: Removed LNI 192.168.201.152@tcp30 [15530.357402] LNet: Removed LNI 192.168.201.152@tcp31 [15530.361036] LNet: Removed LNI 192.168.201.152@tcp32 [15530.363802] LNet: Removed LNI 192.168.201.152@tcp33 [15530.366378] LNet: Removed LNI 192.168.201.152@tcp34 [15530.369236] LNet: Removed LNI 192.168.201.152@tcp35 [15530.372174] LNet: Removed LNI 192.168.201.152@tcp36 [15530.373791] LNet: Removed LNI 192.168.201.152@tcp37 [15530.375817] LNet: Removed LNI 192.168.201.152@tcp38 [15530.378369] LNet: Removed LNI 192.168.201.152@tcp39 [15530.380153] LNet: Removed LNI 192.168.201.152@tcp40 [15530.385342] LNet: Removed LNI 192.168.201.152@tcp41 [15530.387748] LNet: Removed LNI 192.168.201.152@tcp42 [15530.390078] LNet: Removed LNI 192.168.201.152@tcp43 [15530.391931] LNet: Removed LNI 192.168.201.152@tcp44 [15530.393918] LNet: Removed LNI 192.168.201.152@tcp45 [15530.396260] LNet: Removed LNI 192.168.201.152@tcp46 [15530.398562] LNet: Removed LNI 192.168.201.152@tcp47 [15530.401275] LNet: Removed LNI 192.168.201.152@tcp48 [15530.403596] LNet: Removed LNI 192.168.201.152@tcp49 [15530.411417] LNet: Removed LNI 192.168.201.152@tcp50 [15531.367565] Key type .llcrypt unregistered [15531.375384] Key type ._llcrypt unregistered [15554.822152] Key type ._llcrypt registered [15554.823735] Key type .llcrypt registered [15554.972452] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [15571.893740] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15573.057416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15573.115413] alg: No test for adler32 (adler32-zlib) [15574.219320] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [15574.561678] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [15576.295229] Key type lgssc registered [15577.271693] Lustre: Echo OBD driver; http://www.lustre.org/ [15615.372595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15615.390043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15616.725157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15616.766871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15616.881486] Lustre: lustre-MDT0000: new disk, initializing [15616.984783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15617.020104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15621.374797] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15634.003824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15634.163947] Lustre: 506146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15634.248477] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15634.260313] Lustre: Skipped 1 previous similar message [15634.359890] Lustre: lustre-MDT0001: new disk, initializing [15634.444277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15634.478119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15634.487909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15639.145611] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15650.108297] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15656.841866] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15668.098682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15668.348826] Lustre: *** cfs_fail_loc=90f, val=0*** [15668.588363] Lustre: lustre-OST0000: new disk, initializing [15668.599514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15668.606660] Lustre: 508162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15668.758178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15677.094872] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15678.048074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15678.067596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15678.249053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15689.233613] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15704.032924] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15704.033578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15704.054672] LustreError: Skipped 1 previous similar message [15704.058361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15704.086276] Lustre: Skipped 1 previous similar message [15704.121723] Lustre: Skipped 1 previous similar message [15709.173149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15709.178806] Lustre: Skipped 1 previous similar message [15714.278843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15714.282866] Lustre: Skipped 1 previous similar message [15715.298551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15715.441128] Lustre: server umount lustre-OST0000 complete [15721.441202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15721.457412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15721.504890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15724.530300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15725.883487] Lustre: server umount lustre-MDT0000 complete [15729.143673] LustreError: 508161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786767244 with bad export cookie 9841471214331252345 [15729.146519] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15729.154856] LustreError: 508161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15729.871182] Lustre: server umount lustre-MDT0001 complete [15742.477489] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [15745.776550] Key type lgssc unregistered [15746.202918] LNet: 510072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15746.209280] LNetError: 510072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15747.237466] LNet: Removed LNI 192.168.201.152@tcp [15748.591342] Key type .llcrypt unregistered [15748.597722] Key type ._llcrypt unregistered [15770.508982] Key type ._llcrypt registered [15770.511895] Key type .llcrypt registered [15770.608481] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [15771.591308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15771.626077] alg: No test for adler32 (adler32-zlib) [15772.721500] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [15773.083394] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [15774.743161] Key type lgssc registered [15776.281359] Lustre: Echo OBD driver; http://www.lustre.org/ [15786.718989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15786.749131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15788.204202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15793.869134] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15798.766656] LustreError: 511347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15798.785570] LustreError: 511347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15803.886328] LustreError: 511348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15804.954243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15805.514421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15811.044075] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15820.348406] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15826.933354] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15834.595638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15835.086166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15842.316622] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15851.278125] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15877.223049] loop0: detected capacity change from 0 to 204800000 [15877.479796] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15877.502067] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15877.547118] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15885.749412] loop0: detected capacity change from 0 to 204800000 [15886.059949] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15886.073537] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15886.082151] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15891.933934] loop0: detected capacity change from 0 to 204800000 [15898.831504] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15899.134605] Lustre: fs15246-OST0000: new disk, initializing [15899.140837] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15899.149852] Lustre: 514963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15899.228091] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15906.720958] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15919.186888] loop1: detected capacity change from 0 to 204800000 [15926.753314] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15926.969496] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15927.094254] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15927.188769] Lustre: fs15246-MDT0000: new disk, initializing [15927.415118] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15927.449426] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15927.605133] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15928.679273] Lustre: 516088:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000051641306 x1873561295838208/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786767459 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15933.795304] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15933.809181] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15933.850800] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15934.393864] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [15959.015443] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15959.035548] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15964.147347] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15965.451542] Lustre: server umount fs15246-MDT0000 complete [15969.257941] LustreError: 512443:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15974.373233] LustreError: 512443:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15976.699706] Lustre: server umount fs15246-OST0000 complete [15991.778232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15991.790827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15991.820911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15993.825077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15993.839279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15993.858103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15997.580332] Lustre: server umount lustre-OST0000 complete [16005.089654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16005.090671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16005.090795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16005.090800] Lustre: Skipped 2 previous similar messages [16005.108190] Lustre: Skipped 1 previous similar message [16008.354469] Lustre: server umount lustre-MDT0000 complete [16010.214280] LustreError: 512290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16010.233160] LustreError: 512290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16012.733678] LustreError: 516082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786767527 with bad export cookie 9615340344661747010 [16012.749893] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16012.754541] LustreError: 516082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16013.205800] Lustre: server umount lustre-MDT0001 complete [16027.575362] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [16030.707770] Key type lgssc unregistered [16031.048834] LNet: 519187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16031.062195] LNetError: 519187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16031.083257] LNet: Removed LNI 192.168.201.152@tcp [16032.549396] Key type .llcrypt unregistered [16032.551635] Key type ._llcrypt unregistered [16059.443582] Key type ._llcrypt registered [16059.445677] Key type .llcrypt registered [16059.535294] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16061.204243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16061.335724] alg: No test for adler32 (adler32-zlib) [16062.962913] Lustre: Lustre: Build Version: 2.17.57_4_gb324e6e [16063.427465] LNet: Added LNI 192.168.201.152@tcp [8/256/0/180] [16065.327180] Key type lgssc registered [16067.629381] Lustre: Echo OBD driver; http://www.lustre.org/ [16112.075653] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [16121.521163] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16162.692211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16162.740277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16164.138627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16164.200731] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16164.350585] Lustre: lustre-MDT0000: new disk, initializing [16164.519815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16164.603840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16169.853297] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16181.989553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16182.204394] Lustre: 523656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16182.276774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16182.287544] Lustre: Skipped 1 previous similar message [16182.393956] Lustre: lustre-MDT0001: new disk, initializing [16182.594271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16182.664881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16182.713956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16188.285042] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16198.947669] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16207.052723] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16216.048844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16216.473948] Lustre: lustre-OST0000: new disk, initializing [16216.477236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16216.487403] Lustre: 525628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16216.614066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16223.831872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16223.840666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16224.036144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16224.211705] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_default_debug -1 all [16235.134202] Lustre: DEBUG MARKER: oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16241.834366] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16242.215046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16247.183778] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16247.434293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16249.838468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.850551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16249.859225] Lustre: Skipped 1 previous similar message [16249.866709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16249.894822] LustreError: Skipped 1 previous similar message [16254.953081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16254.957698] Lustre: Skipped 1 previous similar message [16260.067984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16260.080396] Lustre: Skipped 2 previous similar messages [16264.159482] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16264.358949] Lustre: server umount lustre-OST0000 complete [16269.794738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16269.808254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16269.828352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16275.139121] Lustre: server umount lustre-MDT0000 complete [16279.641196] LustreError: 523648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786767794 with bad export cookie 10491166706728638521 [16279.648700] LustreError: MGC192.168.201.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16279.658735] LustreError: 523648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16280.193498] Lustre: server umount lustre-MDT0001 complete [16311.195730] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing set_hostid [16322.861467] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing load_modules_local [16361.130929] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16085 sec ========= 00:24:33 (1786767873) [16362.847688] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:24:35 (1786767875) === [16366.501912] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:24:39 (1786767879) === [16410.091499] Lustre: DEBUG MARKER: oleg152-server.virtnet: executing unload_modules_local [16413.308553] Key type lgssc unregistered [16413.654480] LNet: 532279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16413.675731] LNetError: 532279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16414.707287] LNet: Removed LNI 192.168.201.152@tcp [16415.976736] Key type .llcrypt unregistered [16415.978954] Key type ._llcrypt unregistered