[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 577008744 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003280] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007860] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011039] Yama: becoming mindful. [ 0.012057] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.022350] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027389] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032145] *** VALIDATE tmpfs *** [ 0.033531] *** VALIDATE proc *** [ 0.034300] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.036305] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038068] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040045] Spectre V2 : User space: Vulnerable [ 0.041014] Speculative Store Bypass: Vulnerable [ 0.044743] debug: unmapping init [mem 0xffffffffad259000-0xffffffffad260fff] [ 0.047233] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048648] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051010] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053025] ... max period: 00007fffffffffff [ 0.054025] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.056332] rcu: Hierarchical SRCU implementation. [ 0.058710] smp: Bringing up secondary CPUs ... [ 0.059590] x86: Booting SMP configuration: [ 0.060029] .... node #0, CPUs: #1 #2 #3 [ 0.066848] smp: Brought up 1 node, 4 CPUs [ 0.067951] smpboot: Max logical packages: 1 [ 0.068012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128065] node 0 deferred pages initialised in 58ms [ 0.131306] devtmpfs: initialized [ 0.132158] x86/mm: Memory block size: 128MB [ 0.134433] gcov: version magic: 0x41383552 [ 0.136057] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141547] pinctrl core: initialized pinctrl subsystem [ 0.143194] [ 0.143523] ************************************************************* [ 0.145012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146012] ** ** [ 0.147053] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149045] ** ** [ 0.151016] ** This means that this kernel is built to expose internal ** [ 0.152009] ** IOMMU data structures, which may compromise security on ** [ 0.154012] ** your system. ** [ 0.156016] ** ** [ 0.158058] ** If you see this message and you are not debugging the ** [ 0.160007] ** kernel, report this immediately to your vendor! ** [ 0.161016] ** ** [ 0.162011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164007] ************************************************************* [ 0.165610] NET: Registered protocol family 16 [ 0.166384] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.170076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173298] cpuidle: using governor menu [ 0.175267] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176357] PCI: Using configuration type 1 for base access [ 0.178118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.188234] cryptd: max_cpu_qlen set to 1000 [ 0.191231] ACPI: Added _OSI(Module Device) [ 0.193047] ACPI: Added _OSI(Processor Device) [ 0.194008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198507] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203410] ACPI: Interpreter enabled [ 0.205058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206009] ACPI: Using IOAPIC for interrupt routing [ 0.207157] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209309] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218484] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.221049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.230527] acpiphp: Slot [2] registered [ 0.231099] acpiphp: Slot [5] registered [ 0.232065] acpiphp: Slot [6] registered [ 0.233011] acpiphp: Slot [7] registered [ 0.234080] acpiphp: Slot [8] registered [ 0.235065] acpiphp: Slot [9] registered [ 0.236034] acpiphp: Slot [10] registered [ 0.237185] acpiphp: Slot [3] registered [ 0.239097] acpiphp: Slot [4] registered [ 0.240116] acpiphp: Slot [11] registered [ 0.242163] acpiphp: Slot [12] registered [ 0.244148] acpiphp: Slot [13] registered [ 0.245077] acpiphp: Slot [14] registered [ 0.246065] acpiphp: Slot [15] registered [ 0.247074] acpiphp: Slot [16] registered [ 0.248093] acpiphp: Slot [17] registered [ 0.249115] acpiphp: Slot [18] registered [ 0.250110] acpiphp: Slot [19] registered [ 0.251052] acpiphp: Slot [20] registered [ 0.252037] acpiphp: Slot [21] registered [ 0.252964] acpiphp: Slot [22] registered [ 0.254059] acpiphp: Slot [23] registered [ 0.255058] acpiphp: Slot [24] registered [ 0.257117] acpiphp: Slot [25] registered [ 0.258092] acpiphp: Slot [26] registered [ 0.260109] acpiphp: Slot [27] registered [ 0.261095] acpiphp: Slot [28] registered [ 0.262095] acpiphp: Slot [29] registered [ 0.263115] acpiphp: Slot [30] registered [ 0.265099] acpiphp: Slot [31] registered [ 0.266095] PCI host bridge to bus 0000:00 [ 0.267019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.276016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277165] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.278766] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.280873] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.288017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.290089] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.294018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.296426] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.298938] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.302080] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.306015] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.311025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.325941] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329961] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334354] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.340024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.345030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.362030] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.372108] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.379083] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.387030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.406033] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421730] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.429026] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.438024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.458030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.469018] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.477032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.485033] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.505035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.519726] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.527019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.534029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.550020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.561523] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.570030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.579030] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.610033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.620639] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.625487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.627470] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628414] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.631232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.636627] iommu: Default domain type: Passthrough [ 0.637416] SCSI subsystem initialized [ 0.638156] ACPI: bus type USB registered [ 0.640168] usbcore: registered new interface driver usbfs [ 0.641071] usbcore: registered new interface driver hub [ 0.642071] usbcore: registered new device driver usb [ 0.644175] pps_core: LinuxPPS API ver. 1 registered [ 0.645006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.648066] PTP clock support registered [ 0.649156] EDAC MC: Ver: 3.0.0 [ 0.650350] PCI: Using ACPI for IRQ routing [ 0.653030] NetLabel: Initializing [ 0.654015] NetLabel: domain hash size = 128 [ 0.656016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.658142] NetLabel: unlabeled traffic allowed by default [ 0.661150] vgaarb: loaded [ 0.662233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.664008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.673200] clocksource: Switched to clocksource kvm-clock [ 0.781950] VFS: Disk quotas dquot_6.6.0 [ 0.783255] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.784832] *** VALIDATE ramfs *** [ 0.785620] *** VALIDATE hugetlbfs *** [ 0.786762] pnp: PnP ACPI init [ 0.788617] pnp: PnP ACPI: found 6 devices [ 0.804100] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.806582] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.808482] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.809896] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.811539] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.813332] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.815489] NET: Registered protocol family 2 [ 0.817377] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.821744] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.826074] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.832235] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.836570] TCP: Hash tables configured (established 65536 bind 65536) [ 0.840105] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.843469] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846712] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849852] NET: Registered protocol family 1 [ 0.852894] RPC: Registered named UNIX socket transport module. [ 0.855096] RPC: Registered udp transport module. [ 0.857051] RPC: Registered tcp transport module. [ 0.859160] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861807] NET: Registered protocol family 44 [ 0.863253] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.864454] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.865829] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.867257] PCI: CLS 0 bytes, default 64 [ 0.868321] Unpacking initramfs... [ 2.311332] debug: unmapping init [mem 0xffff94a4bcc54000-0xffff94a4bffbffff] [ 2.314838] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.317016] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.320216] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.810810] Initialise system trusted keyrings [ 2.812679] Key type blacklist registered [ 2.814860] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.822214] zbud: loaded [ 2.825495] *** VALIDATE nfs *** [ 2.826634] *** VALIDATE nfs4 *** [ 2.828481] pstore: using deflate compression [ 2.832043] Platform Keyring initialized [ 2.949303] NET: Registered protocol family 38 [ 2.951187] Key type asymmetric registered [ 2.953030] Asymmetric key parser 'x509' registered [ 2.955666] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.961552] io scheduler mq-deadline registered [ 2.963656] io scheduler kyber registered [ 2.965846] io scheduler bfq registered [ 2.968117] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.971439] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.974098] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.976487] ACPI: Power Button [PWRF] [ 2.981048] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.986600] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.997882] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.003087] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.016508] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.042186] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.071594] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.077503] Non-volatile memory driver v1.3 [ 3.079260] Linux agpgart interface v0.103 [ 3.114941] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.117330] vda: detected capacity change from 0 to 74698752 [ 3.132923] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.135028] vdb: detected capacity change from 0 to 1073741824 [ 3.150484] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.153198] vdc: detected capacity change from 0 to 2621440000 [ 3.169722] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.173063] vdd: detected capacity change from 0 to 2621440000 [ 3.187622] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.190121] vde: detected capacity change from 0 to 4294967296 [ 3.204483] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.207758] vdf: detected capacity change from 0 to 4294967296 [ 3.221361] libphy: Fixed MDIO Bus: probed [ 3.241332] usbcore: registered new interface driver usbserial_generic [ 3.243326] usbserial: USB Serial support registered for generic [ 3.245085] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.248251] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.249388] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.251222] mousedev: PS/2 mouse device common for all mice [ 3.253480] rtc_cmos 00:05: RTC can wake from S4 [ 3.255583] rtc_cmos 00:05: registered as rtc0 [ 3.257176] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.259366] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.261846] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.266812] intel_pstate: CPU model not supported [ 3.271051] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.274036] hid: raw HID events driver (C) Jiri Kosina [ 3.275520] usbcore: registered new interface driver usbhid [ 3.276946] usbhid: USB HID core driver [ 3.278120] drop_monitor: Initializing network drop monitor service [ 3.279768] Initializing XFRM netlink socket [ 3.281056] NET: Registered protocol family 10 [ 3.285188] Segment Routing with IPv6 [ 3.286594] NET: Registered protocol family 17 [ 3.288324] mpls_gso: MPLS GSO support [ 3.293396] RAS: Correctable Errors collector initialized. [ 3.295001] AVX version of gcm_enc/dec engaged. [ 3.296120] AES CTR mode by8 optimization enabled [ 3.373965] sched_clock: Marking stable (3373921224, 0)->(4299868353, -925947129) [ 3.376793] registered taskstats version 1 [ 3.378723] Loading compiled-in X.509 certificates [ 3.380855] zswap: loaded using pool lzo/zbud [ 3.406502] Key type big_key registered [ 3.418846] Key type encrypted registered [ 3.420119] ima: No TPM chip found, activating TPM-bypass! [ 3.422024] ima: Allocated hash algorithm: sha1 [ 3.423565] ima: No architecture policies found [ 3.425503] evm: Initialising EVM extended attributes: [ 3.427623] evm: security.selinux [ 3.429211] evm: security.ima [ 3.430309] evm: security.capability [ 3.431640] evm: HMAC attrs: 0x1 [ 3.435900] rtc_cmos 00:05: setting system clock to 2026-08-14 06:31:16 UTC (1786689076) [ 3.442155] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 3.445311] debug: unmapping init [mem 0xffffffffacf82000-0xffffffffad258fff] [ 3.454116] Write protecting the kernel read-only data: 28672k [ 3.457756] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 3.460638] debug: unmapping init [mem 0xffffffffabf14000-0xffffffffabffffff] [ 3.502753] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.509045] systemd[1]: Detected virtualization kvm. [ 3.511077] systemd[1]: Detected architecture x86-64. [ 3.513209] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.540552] systemd[1]: No hostname configured. [ 3.542612] systemd[1]: Set hostname to . [ 3.544844] random: systemd: uninitialized urandom read (16 bytes read) [ 3.547435] systemd[1]: Initializing machine ID from random generator. [ 3.588567] random: ln: uninitialized urandom read (6 bytes read) [ 3.683230] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685301] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.688385] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.691632] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.308856] device-mapper: uevent: version 1.0.3 [ 4.310269] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.072430] virtio_net virtio0 ens2: renamed from eth0 [ 5.106784] random: fast init done [ 5.197904] scsi host0: ata_piix [ 5.224260] scsi host1: ata_piix [ 5.225830] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.228341] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.614796] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.939643] random: crng init done [ 9.941176] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.442494] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.623997] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.896297] SELinux: Disabled at runtime. [ 11.967055] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.976453] systemd[1]: Detected virtualization kvm. [ 11.978425] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.513350] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.516186] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.522370] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.527466] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.531584] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.540196] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.544536] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.645792] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.980176] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.374439] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.401303] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.526591] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.540186] EDAC sbridge: Ver: 1.1.2 [ 15.193481] Key type dns_resolver registered [ 15.504290] NFS: Registering the id_resolver key type [ 15.506368] Key type id_resolver registered [ 15.508031] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg206-server login: [ 41.632552] libcfs: loading out-of-tree module taints kernel. [ 41.654801] Key type ._llcrypt registered [ 41.656632] Key type .llcrypt registered [ 41.712543] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 50.237417] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 50.854968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.863641] alg: No test for adler32 (adler32-zlib) [ 51.900478] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 52.239308] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 53.871174] Key type lgssc registered [ 54.682765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.480131] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 65.658012] hrtimer: interrupt took 2098061 ns [ 110.622788] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 124.714162] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 124.743558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.038690] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 126.085700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 126.242532] Lustre: lustre-MDT0000: new disk, initializing [ 126.431135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.472447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 131.639798] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 147.905869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.083532] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 148.152141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 148.163697] Lustre: Skipped 1 previous similar message [ 148.284250] Lustre: lustre-MDT0001: new disk, initializing [ 148.365478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 148.410823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 148.424701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 152.804876] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.560210] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.688209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.983542] Lustre: lustre-OST0000: new disk, initializing [ 167.987562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.998343] Lustre: 8419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 168.057983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.642252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 171.655545] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 171.841741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.722422] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.577830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 192.721469] Lustre: lustre-OST0001: new disk, initializing [ 192.725774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 192.743853] Lustre: 9492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.830732] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.060310] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.797812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 201.823026] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 201.933575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.924209] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.819990] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.844383] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing check_logdir /tmp/testlogs/ [ 231.661605] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing yml_node [ 237.408950] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 241.229383] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 244.576892] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 246.522778] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 02:35:17 EDT 2026 [ 264.946594] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 266.778604] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 284.129723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.143954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 284.163971] Lustre: Skipped 2 previous similar messages [ 284.190710] Lustre: Skipped 3 previous similar messages [ 288.588639] Lustre: server umount lustre-MDT0000 complete [ 293.026359] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786689366 with bad export cookie 11504312642994687179 [ 293.036223] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.039391] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 294.367834] LustreError: 10726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 294.369721] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.376846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 294.398479] LustreError: 10726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 294.435622] Lustre: Skipped 2 previous similar messages [ 298.400840] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.401064] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.413191] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 298.424359] Lustre: Skipped 2 previous similar messages [ 299.522547] Lustre: server umount lustre-MDT0001 complete [ 303.858522] Lustre: server umount lustre-OST0000 complete [ 307.641367] Lustre: server umount lustre-OST0001 complete [ 314.338525] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 325.629661] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 366.842379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.136720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 367.164184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 367.232388] Lustre: lustre-MDT0000: new disk, initializing [ 367.363741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 367.387821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 371.817915] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 384.292982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.476867] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 384.532924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 384.536899] Lustre: Skipped 1 previous similar message [ 384.635704] Lustre: lustre-MDT0001: new disk, initializing [ 384.825454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 384.895226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 384.908105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.684995] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 401.552678] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 409.059783] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 415.981180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.292184] Lustre: lustre-OST0000: new disk, initializing [ 416.298575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 416.304968] Lustre: 17019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 416.397120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.227679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 418.239931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 418.452112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.869727] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 433.379693] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 438.910949] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 439.287324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 443.699133] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 443.918283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 448.995738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.997703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 448.999676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.012239] Lustre: Skipped 1 previous similar message [ 449.033756] Lustre: Skipped 1 previous similar message [ 452.161344] Lustre: server umount lustre-OST0000 complete [ 456.674334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 456.687736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.715763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 464.354133] LustreError: 16613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.386540] LustreError: 16613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 464.648401] Lustre: server umount lustre-MDT0000 complete [ 469.205861] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786689542 with bad export cookie 11504312642994688712 [ 469.211174] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.217877] LustreError: 15043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 469.597891] Lustre: server umount lustre-MDT0001 complete [ 473.872747] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:39:05 (1786689545) [ 476.482577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 478.432923] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:39:10 (1786689550) [ 484.656358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 485.223945] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.331818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.474302] LustreError: 19432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.581546] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 499.017485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.287159] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 512.639933] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 518.984690] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.092264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.930995] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 542.713728] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 557.544172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.550808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.566165] Lustre: Skipped 1 previous similar message [ 557.577838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.585492] Lustre: Skipped 2 previous similar messages [ 560.609650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 563.279735] Lustre: server umount lustre-OST0000 complete [ 570.853461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.861091] Lustre: Skipped 1 previous similar message [ 573.383132] Lustre: server umount lustre-MDT0000 complete [ 575.969203] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.009517] LustreError: 19431:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 576.732067] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786689649 with bad export cookie 11504312642994689594 [ 576.756363] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.761358] LustreError: 19413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 577.185840] Lustre: server umount lustre-MDT0001 complete [ 589.311887] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 592.196406] Key type lgssc unregistered [ 592.521083] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.530547] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.546442] LNet: Removed LNI 192.168.202.106@tcp [ 593.357137] Key type .llcrypt unregistered [ 593.358759] Key type ._llcrypt unregistered [ 618.584348] Key type ._llcrypt registered [ 618.586378] Key type .llcrypt registered [ 618.672579] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 619.663568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 619.729427] alg: No test for adler32 (adler32-zlib) [ 621.060654] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 621.314329] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 622.999183] Key type lgssc registered [ 624.470111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.057313] loop: module loaded [ 634.150575] loop0: detected capacity change from 0 to 204800000 [ 634.492308] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.503545] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 634.848975] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.193244] loop0: detected capacity change from 0 to 204800000 [ 641.543516] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.560132] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 641.582125] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 647.873614] loop0: detected capacity change from 0 to 204800000 [ 655.947996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.002905] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.411515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 657.489211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 657.645639] Lustre: lustre-MDT0000: new disk, initializing [ 657.786084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 657.836335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 663.211527] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 672.032351] loop1: detected capacity change from 0 to 204800000 [ 677.754044] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 678.016320] Lustre: lustre-OST07c6: new disk, initializing [ 678.019791] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 678.023754] Lustre: Skipped 1 previous similar message [ 678.031635] Lustre: 26169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 678.095301] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 684.730801] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 686.757215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 686.764667] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 686.819583] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 690.549641] Lustre: 25157:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 690.559955] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 690.565354] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 690.571272] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 690.575996] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 690.581885] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 706.974557] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 706.986351] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.992670] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.999522] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 707.008872] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 707.013991] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 712.900140] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 712.918454] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 712.931079] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.943080] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.952995] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.964152] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 712.976413] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 713.006577] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.018537] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 713.031772] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 713.041388] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 713.058351] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.370761] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 716.381813] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 716.398203] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 716.401730] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.406484] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.410906] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.415889] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 716.423784] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.427760] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.432232] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.437441] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.440896] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.815255] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 721.820596] Lustre: 26177:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 721.825036] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 721.828861] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.832331] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 721.836107] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.841274] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 721.845391] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.848564] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 721.852866] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 721.856154] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 721.860558] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.435927] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 730.450965] Lustre: 25156:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 730.457192] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 730.468255] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.477844] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.489180] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.502197] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 730.508668] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.518664] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.527476] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 730.534341] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.542027] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 782.597169] Lustre: server umount lustre-OST07c6 complete [ 803.453756] Lustre: server umount lustre-MDT0000 complete [ 821.298462] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 824.713482] Key type lgssc unregistered [ 825.049988] LNet: 28732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.065840] LNetError: 28732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.092139] LNet: Removed LNI 192.168.202.106@tcp [ 826.834614] Key type .llcrypt unregistered [ 826.837949] Key type ._llcrypt unregistered [ 845.569184] Key type ._llcrypt registered [ 845.570562] Key type .llcrypt registered [ 845.730784] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 847.301211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.320585] alg: No test for adler32 (adler32-zlib) [ 848.326864] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 848.532768] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 850.215181] Key type lgssc registered [ 851.424965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 859.834559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 859.847130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.242694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.093630] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 871.397398] LustreError: 30000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.430593] LustreError: 30000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 876.414518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 876.512841] LustreError: 30001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.977503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 882.098484] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 890.248669] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 895.484534] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 902.876337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 903.274084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.386017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 909.971084] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 918.179128] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 942.573259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.577139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 942.608204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 942.619641] LustreError: Skipped 1 previous similar message [ 942.646827] Lustre: Skipped 1 previous similar message [ 947.700399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.711810] Lustre: Skipped 1 previous similar message [ 948.856962] Lustre: server umount lustre-OST0000 complete [ 953.839433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 953.858926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.893834] Lustre: Skipped 1 previous similar message [ 953.908372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.920815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.922784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 957.958112] Lustre: Skipped 1 previous similar message [ 960.009229] Lustre: server umount lustre-MDT0000 complete [ 963.047386] LustreError: 29997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.063562] LustreError: 29997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 964.446259] LustreError: 30734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786690037 with bad export cookie 538234119565412486 [ 964.447222] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.457573] LustreError: 30734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 965.114484] Lustre: server umount lustre-MDT0001 complete [ 978.178279] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 981.606934] Key type lgssc unregistered [ 982.018716] LNet: 33745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.027497] LNetError: 33745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.063699] LNet: Removed LNI 192.168.202.106@tcp [ 983.148372] Key type .llcrypt unregistered [ 983.149577] Key type ._llcrypt unregistered [ 1005.956687] Key type ._llcrypt registered [ 1005.959862] Key type .llcrypt registered [ 1006.104880] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1007.505626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.559203] alg: No test for adler32 (adler32-zlib) [ 1008.918269] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1009.272970] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1011.040075] Key type lgssc registered [ 1012.633701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1022.066601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1022.122845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.810152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1029.098877] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1033.201258] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.224723] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1038.314047] LustreError: 35019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.094713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.914218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.243804] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1057.647216] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1065.733956] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1075.507280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.090056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.466099] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1092.153235] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1093.514264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1108.963160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1108.974410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.995545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1109.032410] Lustre: Skipped 1 previous similar message [ 1112.319456] Lustre: server umount lustre-OST0000 complete [ 1119.203477] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.206618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1119.227423] Lustre: Skipped 1 previous similar message [ 1119.230381] Lustre: Skipped 1 previous similar message [ 1124.322727] LustreError: 35014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1124.356957] LustreError: 35014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1124.383965] Lustre: server umount lustre-MDT0000 complete [ 1129.118446] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786690202 with bad export cookie 13381427105028074734 [ 1129.120624] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.137273] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1129.424052] Lustre: server umount lustre-MDT0001 complete [ 1144.515433] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1147.973847] Key type lgssc unregistered [ 1148.387549] LNet: 38573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.393965] LNetError: 38573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.418562] LNet: Removed LNI 192.168.202.106@tcp [ 1150.542093] Key type .llcrypt unregistered [ 1150.544448] Key type ._llcrypt unregistered [ 1178.578975] Key type ._llcrypt registered [ 1178.584374] Key type .llcrypt registered [ 1178.753376] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1180.171174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1180.221299] alg: No test for adler32 (adler32-zlib) [ 1181.557701] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1181.991549] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1183.759568] Key type lgssc registered [ 1185.874418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.219088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.246638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.801731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.699920] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1207.271271] LustreError: 39994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.298798] LustreError: 39994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1212.390484] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.269505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.024852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.257655] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1228.239499] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1235.226358] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1242.665721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.056729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.075733] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1260.642233] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1267.640101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1277.926725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1277.936327] Lustre: Skipped 2 previous similar messages [ 1277.939204] LustreError: 39988:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.077179] Lustre: server umount lustre-MDT0000 complete [ 1282.147750] LustreError: 39989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.167905] LustreError: 39989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1282.395445] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786690355 with bad export cookie 16603989970584253093 [ 1282.397208] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1282.403543] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1283.054074] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.075892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1284.758586] Lustre: server umount lustre-MDT0001 complete [ 1297.300892] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.6@tcp arrived at 1786690370 with bad export cookie 16603989970584253758 [ 1297.313172] LustreError: 39974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1315.296125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1315.758483] Lustre: server umount lustre-OST0000 complete [ 1336.623867] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1340.127136] Key type lgssc unregistered [ 1340.489832] LNet: 43887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.495381] LNetError: 43887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.537106] LNet: Removed LNI 192.168.202.106@tcp [ 1341.694641] Key type .llcrypt unregistered [ 1341.696269] Key type ._llcrypt unregistered [ 1370.792307] Key type ._llcrypt registered [ 1370.794033] Key type .llcrypt registered [ 1370.986907] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1372.776285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.970647] alg: No test for adler32 (adler32-zlib) [ 1374.280989] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1374.604280] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1376.311699] Key type lgssc registered [ 1377.788455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.346761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.363455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.836285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.507094] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1399.266336] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.296385] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1404.391922] LustreError: 45307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.100465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.831444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1413.587616] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1423.022815] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.899225] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1441.188576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.582416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.698918] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1459.020803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1459.155843] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1466.468210] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1468.393693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.406441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.419842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.456814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.457365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.467806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.499348] Lustre: Skipped 1 previous similar message [ 1471.374866] Lustre: server umount lustre-OST0000 complete [ 1489.888513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.891686] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.918828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.941791] Lustre: Skipped 1 previous similar message [ 1494.052780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1494.064052] Lustre: Skipped 1 previous similar message [ 1504.225321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1504.230039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.264162] Lustre: Skipped 3 previous similar messages [ 1504.463426] Lustre: server umount lustre-MDT0000 complete [ 1509.353269] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.378763] LustreError: 45306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1509.634764] LustreError: 46043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786690582 with bad export cookie 3451934546740939481 [ 1509.641882] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.649690] LustreError: 46043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1516.260116] Lustre: server umount lustre-MDT0001 complete [ 1533.086916] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1536.469285] Key type lgssc unregistered [ 1536.764401] LNet: 49028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.776177] LNetError: 49028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.790727] LNet: Removed LNI 192.168.202.106@tcp [ 1537.729087] Key type .llcrypt unregistered [ 1537.730948] Key type ._llcrypt unregistered [ 1566.949282] Key type ._llcrypt registered [ 1566.952596] Key type .llcrypt registered [ 1567.142621] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1569.015608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.050894] alg: No test for adler32 (adler32-zlib) [ 1570.377896] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1570.793945] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1572.463368] Key type lgssc registered [ 1573.926192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1583.507861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1583.551567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.030138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.387557] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1595.362410] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.402236] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1600.496091] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.956218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.340628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.393234] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1616.074386] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1623.908827] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1632.095560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.635130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1638.678588] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1647.135103] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1655.169506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1663.799506] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1667.606183] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1680.917610] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-14 2h59m11s [ 1682.544639] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1685.984708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.990052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1686.000979] Lustre: Skipped 2 previous similar messages [ 1686.015729] Lustre: Skipped 2 previous similar messages [ 1687.086403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.102469] Lustre: Skipped 2 previous similar messages [ 1689.184671] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 1693.154517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.170333] Lustre: Skipped 3 previous similar messages [ 1699.300699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.302223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 1699.309807] Lustre: Skipped 3 previous similar messages [ 1699.494828] Lustre: server umount lustre-MDT0000 complete [ 1702.470518] LustreError: 50441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.403926] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1705.451932] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.482270] LustreError: 50447:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1711.595066] LustreError: 51175:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.621738] LustreError: 51175:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1713.498903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.859460] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.131347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.192704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1715.191467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1719.967152] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1725.047735] Lustre: DEBUG MARKER: Wait for df (36725) ... [ 1726.941102] Lustre: DEBUG MARKER: done [ 1730.544408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1730.570394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1730.590357] Lustre: Skipped 1 previous similar message [ 1744.352986] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1744.483896] Lustre: server umount lustre-OST0000 complete [ 1751.015457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.019800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.038037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.038045] Lustre: Skipped 4 previous similar messages [ 1751.052744] Lustre: Skipped 2 previous similar messages [ 1763.296402] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1763.604582] Lustre: server umount lustre-MDT0000 complete [ 1764.328343] LustreError: 50441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1764.354304] LustreError: 50441:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1768.012221] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786690841 with bad export cookie 5679460622018820229 [ 1768.024245] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.040136] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1768.458291] Lustre: server umount lustre-MDT0001 complete [ 1781.366176] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 1784.613129] Key type lgssc unregistered [ 1784.960066] LNet: 55402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.966579] LNetError: 55402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.026460] LNet: Removed LNI 192.168.202.106@tcp [ 1787.384500] Key type .llcrypt unregistered [ 1787.393897] Key type ._llcrypt unregistered [ 1836.254211] Key type ._llcrypt registered [ 1836.255996] Key type .llcrypt registered [ 1836.407378] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 1837.948809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.009360] alg: No test for adler32 (adler32-zlib) [ 1839.057718] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 1839.368831] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 1841.065361] Key type lgssc registered [ 1842.688739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1850.937689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1850.965562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.326727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.358310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.754364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.297070] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1867.991144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.110645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1868.139712] Lustre: 57627:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1868.448147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1873.698424] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1883.502115] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1891.215366] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1899.204367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1899.338090] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1899.522394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.077493] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 1907.891556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1914.714660] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.833581] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-14 3h03m14s [ 1925.274315] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1933.796605] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1937.111187] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1937.198027] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1957.507215] Lustre: lustre-MDT0000: Client 87fa3d0a-0ccb-4127-acce-0b3c7f38a546 (at 192.168.202.6@tcp) reconnecting [ 1977.263124] LustreError: 57636:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1980.891433] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1984.580468] Lustre: DEBUG MARKER: done [ 1994.729126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1994.741655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.765601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.238559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.256478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.277598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1995.284166] Lustre: Skipped 1 previous similar message [ 1997.960859] Lustre: server umount lustre-OST0000 complete [ 2005.481817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.485318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2005.499508] Lustre: Skipped 1 previous similar message [ 2005.515287] Lustre: Skipped 1 previous similar message [ 2008.894140] Lustre: server umount lustre-MDT0000 complete [ 2012.993061] LustreError: 60391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691086 with bad export cookie 17840295637344056802 [ 2012.994185] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2013.005610] LustreError: 60391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2013.029576] LustreError: 56953:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [ 2013.638496] Lustre: server umount lustre-MDT0001 complete [ 2026.187746] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2028.970750] Key type lgssc unregistered [ 2029.286575] LNet: 61969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.308367] LNetError: 61969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2029.346333] LNet: Removed LNI 192.168.202.106@tcp [ 2030.287735] Key type .llcrypt unregistered [ 2030.295704] Key type ._llcrypt unregistered [ 2088.098894] Key type ._llcrypt registered [ 2088.108902] Key type .llcrypt registered [ 2088.269576] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2090.157413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2090.227086] alg: No test for adler32 (adler32-zlib) [ 2091.425777] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 2091.775420] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2093.559752] Key type lgssc registered [ 2094.912200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2098.975716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2098.988313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.352418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2100.402695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2100.604192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.484302] Lustre: Failing over lustre-MDT0000 [ 2101.761873] Lustre: server umount lustre-MDT0000 complete [ 2110.051162] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:06:21 (1786691181) [ 2117.676268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2118.055542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2122.252468] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2131.496307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.617223] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2131.667311] Lustre: 64767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2132.034275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2137.128171] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2145.632889] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2152.328757] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2160.430261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2160.595393] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2160.793617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.728639] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2170.549947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2176.404400] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2182.322446] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2193.377400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2193.382606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.402716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.450577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.460979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2196.498757] Lustre: Skipped 2 previous similar messages [ 2198.378224] Lustre: server umount lustre-MDT0000 complete [ 2202.657200] LustreError: 64760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691275 with bad export cookie 3429544897647187122 [ 2202.669712] LustreError: 64760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2202.670707] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2203.135863] Lustre: server umount lustre-MDT0001 complete [ 2207.144964] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2212.402763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2221.543862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.048472] Lustre: 63694:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691279/real 1786691279] req@00000000d3234e6c x1873481474025856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2222.085878] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2222.093433] Lustre: Skipped 1 previous similar message [ 2223.071403] Lustre: 63692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691280/real 1786691280] req@00000000c65d604a x1873481474025984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2227.168269] Lustre: 63691:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691284/real 1786691284] req@0000000052ff8be8 x1873481474026112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2227.237680] LustreError: 63690:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001d594ba6 x1873481474026880/t0(0) o250->MGC192.168.202.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2227.763144] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.788279] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2227.870241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2227.978070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2231.905954] LustreError: 66619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.931544] LustreError: 68543:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2232.958803] LustreError: 68543:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2233.449964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2233.961224] Lustre: 63693:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691291/real 1786691291] req@0000000030b5ea9d x1873481474026496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2234.005818] Lustre: 63693:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2238.434740] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.450808] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2243.437326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.553981] LustreError: 66619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.573522] LustreError: 66619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2243.747524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2249.199913] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2249.497465] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2251.258802] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2258.150080] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2264.466723] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2274.785291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.797757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.816721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.846112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.854024] Lustre: Skipped 1 previous similar message [ 2281.956298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2281.963458] Lustre: Skipped 2 previous similar messages [ 2287.076394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2287.405061] Lustre: server umount lustre-MDT0000 complete [ 2291.673869] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691364 with bad export cookie 3429544897647189397 [ 2291.686252] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2291.690187] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2291.941635] Lustre: server umount lustre-MDT0001 complete [ 2298.913296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2308.579054] Lustre: 63692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691365/real 1786691365] req@000000005ff4a1fb x1873481474056704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2308.633510] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.669772] Lustre: Skipped 2 previous similar messages [ 2310.364785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2317.582033] LustreError: 66618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.679975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.775138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2318.689491] Lustre: 63692:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786691374/real 1786691374] req@000000003594cd91 x1873481474057216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786691390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2318.703789] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2318.725543] Lustre: 63692:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2324.223827] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2334.177809] LustreError: 71225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.231960] LustreError: 71225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2338.251697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2342.969869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2343.926438] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2352.204725] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2360.495649] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2369.506065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.510524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2369.517559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.517570] Lustre: Skipped 3 previous similar messages [ 2369.524826] Lustre: Skipped 1 previous similar message [ 2373.834698] Lustre: server umount lustre-MDT0000 complete [ 2374.626176] LustreError: 71227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.645304] LustreError: 71227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2378.407620] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691451 with bad export cookie 3429544897647190776 [ 2378.408485] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2378.413697] LustreError: 64762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2378.920442] Lustre: server umount lustre-MDT0001 complete [ 2386.185541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2389.805227] Lustre: DEBUG MARKER: files compared the same [ 2401.452035] Lustre: server umount lustre-OST0000 complete [ 2421.803720] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2424.729121] Key type lgssc unregistered [ 2424.995830] LNet: 74508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.009804] LNetError: 74508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.048084] LNet: Removed LNI 192.168.202.106@tcp [ 2426.067350] Key type .llcrypt unregistered [ 2426.069716] Key type ._llcrypt unregistered [ 2457.945872] Key type ._llcrypt registered [ 2457.952056] Key type .llcrypt registered [ 2458.128705] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2459.892055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2459.907965] alg: No test for adler32 (adler32-zlib) [ 2461.023460] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 2461.224308] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2462.879796] Key type lgssc registered [ 2464.238473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.490837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.513225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.292758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.533175] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2485.736424] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.755430] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2490.852737] LustreError: 75927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.938954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2492.215645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.832320] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2508.065399] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2516.454376] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.131871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.458141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.117303] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug +malloc all [ 2545.971344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2546.274624] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.385586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.417550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.425413] Lustre: Skipped 1 previous similar message [ 2561.507107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.824927] Lustre: server umount lustre-OST0000 complete [ 2569.184316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2569.191080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.202498] Lustre: Skipped 1 previous similar message [ 2569.206147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.208240] Lustre: Skipped 1 previous similar message [ 2571.744467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2571.745286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2571.758393] Lustre: Skipped 1 previous similar message [ 2576.865920] LustreError: 75924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.898724] LustreError: 75924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2577.164602] Lustre: server umount lustre-MDT0000 complete [ 2581.991211] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.018750] LustreError: 75928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2583.626811] LustreError: 75907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691656 with bad export cookie 15059434838873659410 [ 2583.634489] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2583.638935] LustreError: 75907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2584.217243] Lustre: server umount lustre-MDT0001 complete [ 2602.561746] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2605.819040] Key type lgssc unregistered [ 2606.158770] LNet: 79488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.164197] LNetError: 79488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2607.210955] LNet: Removed LNI 192.168.202.106@tcp [ 2608.172071] Key type .llcrypt unregistered [ 2608.173702] Key type ._llcrypt unregistered [ 2641.314789] Key type ._llcrypt registered [ 2641.319687] Key type .llcrypt registered [ 2641.484893] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2644.354925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2644.376154] alg: No test for adler32 (adler32-zlib) [ 2645.550267] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 2645.902787] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2647.920630] Key type lgssc registered [ 2649.484593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2660.087302] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2660.100856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2675.680774] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2680.799554] LustreError: 80880:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [ 2707.551322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.582607] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2726.336875] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2736.041687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.318193] Lustre: *** cfs_fail_loc=706, val=0*** [ 2736.926932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.088742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2742.053689] LustreError: 81969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.072471] LustreError: 81969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2742.667266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2747.364593] LustreError: 81967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.382931] LustreError: 81967:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2752.501874] LustreError: 82414:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2752.528056] LustreError: 82414:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2754.506250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.007679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2760.790027] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2771.203977] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2779.965309] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2783.203158] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.208677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.217675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.762519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.767837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.820139] Lustre: Skipped 1 previous similar message [ 2789.283323] Lustre: server umount lustre-OST0000 complete [ 2796.001992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2796.002565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2796.047257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2796.066198] Lustre: Skipped 1 previous similar message [ 2796.102299] Lustre: Skipped 1 previous similar message [ 2801.127386] LustreError: 81973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.167280] LustreError: 81973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2801.651644] Lustre: server umount lustre-MDT0000 complete [ 2805.228456] LustreError: 81968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2805.257605] LustreError: 81968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2807.615664] LustreError: 83544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786691880 with bad export cookie 15101646650495459863 [ 2807.616708] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2807.641556] LustreError: 83544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2808.449515] Lustre: server umount lustre-MDT0001 complete [ 2829.613648] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 2834.825502] Key type lgssc unregistered [ 2835.386028] LNet: 84526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2835.396551] LNetError: 84526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2835.410953] LNet: Removed LNI 192.168.202.106@tcp [ 2836.364636] Key type .llcrypt unregistered [ 2836.366947] Key type ._llcrypt unregistered [ 2871.578298] Key type ._llcrypt registered [ 2871.583911] Key type .llcrypt registered [ 2871.752766] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 2872.919389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2872.982749] alg: No test for adler32 (adler32-zlib) [ 2874.294966] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 2874.648832] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 2876.311214] Key type lgssc registered [ 2878.243349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.532365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2889.573700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2897.988661] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2907.963995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.913169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.273359] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.298226] LustreError: 86635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2914.678515] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2918.370055] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.407725] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2923.495516] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.514415] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2924.352417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.670821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.151094] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.188717] LustreError: 86638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2932.788372] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2938.339058] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.367227] LustreError: 86639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2943.153931] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.162868] Lustre: MGS: Client 268e316e-205a-49eb-9c26-2aa4a6081fcb (at 0@lo) reconnecting [ 2943.189691] LustreError: 86649:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2943.483599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.566598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2949.796593] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 2968.445124] Lustre: server umount lustre-OST0000 complete [ 2974.179667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.191369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2974.201655] Lustre: Skipped 1 previous similar message [ 2974.211244] Lustre: Skipped 1 previous similar message [ 2979.308575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2979.318458] Lustre: Skipped 1 previous similar message [ 2983.432408] Lustre: server umount lustre-MDT0000 complete [ 2984.422201] LustreError: 88179:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.441111] LustreError: 88179:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2985.439508] Lustre: 85306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786692042/real 1786692042] req@00000000a296b2da x1873482295045248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786692058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2985.480711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.465095] Lustre: 85307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786692043/real 1786692043] req@000000007040befb x1873482295046016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786692059 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2988.489465] Lustre: server umount lustre-MDT0001 complete [ 2993.671659] Lustre: server umount MGS complete [ 3009.027904] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:21:20 (1786692080) [ 3046.331691] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 3058.105616] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3099.754065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.766654] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3116.021969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.309514] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3116.424553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3116.576687] Lustre: lustre-MDT0001: new disk, initializing [ 3116.720840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3116.755528] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3116.765547] Lustre: 93831:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3116.778115] LustreError: 93831:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3122.564497] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3137.073502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.540180] Lustre: lustre-OST0000: new disk, initializing [ 3137.549065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3137.563035] Lustre: 94597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3137.702689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.725596] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3146.274090] Lustre: 94911:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000cfda2b77 x1873482295060992/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786692235 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3156.875338] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3165.572327] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3165.613389] Lustre: MGS: Client e6c42e9d-4bb5-4e84-8edc-c6d7fdf0ed13 (at 0@lo) reconnecting [ 3165.703690] Lustre: 93099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3165.719449] LustreError: 93817:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3165.779364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3165.906167] Lustre: lustre-MDT0000: new disk, initializing [ 3166.141950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.173774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3166.214882] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3167.267695] Lustre: 95666:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000052a70657 x1873482295068288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786692256 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3172.344125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3172.357615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3172.503964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3172.870470] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3202.018064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.021348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3202.034517] Lustre: Skipped 1 previous similar message [ 3202.042346] Lustre: Skipped 1 previous similar message [ 3207.145805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3207.149360] Lustre: Skipped 1 previous similar message [ 3212.263425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3212.271683] Lustre: Skipped 1 previous similar message [ 3214.303247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3214.623478] Lustre: server umount lustre-OST0000 complete [ 3222.500229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3222.502250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.530812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3222.537363] Lustre: Skipped 1 previous similar message [ 3233.247608] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3233.424958] Lustre: server umount lustre-MDT0000 complete [ 3252.191156] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3252.598858] Lustre: server umount lustre-MDT0001 complete [ 3257.380241] Lustre: server umount MGS complete [ 3271.615317] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:25:43 (1786692343) [ 3291.787403] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3294.690511] Key type lgssc unregistered [ 3295.030083] LNet: 98173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3295.051685] LNetError: 98173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.102201] LNet: Removed LNI 192.168.202.106@tcp [ 3297.447658] Key type .llcrypt unregistered [ 3297.450707] Key type ._llcrypt unregistered [ 3317.666395] Key type ._llcrypt registered [ 3317.670264] Key type .llcrypt registered [ 3317.795955] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3319.176562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3319.268929] alg: No test for adler32 (adler32-zlib) [ 3320.550169] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 3320.934602] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3322.671214] Key type lgssc registered [ 3323.871927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3331.482173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3331.494307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.058794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.860241] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3343.332878] LustreError: 99396:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.352265] LustreError: 99396:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3347.950814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3348.524403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3353.836115] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3362.768823] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3370.150687] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3378.270230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3378.636203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3384.388144] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3392.789438] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3403.263264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3413.475135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.475625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.489095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3413.499580] Lustre: Skipped 1 previous similar message [ 3413.521349] LustreError: Skipped 1 previous similar message [ 3417.400454] Lustre: server umount lustre-OST0000 complete [ 3423.713113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.734728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.743224] Lustre: Skipped 2 previous similar messages [ 3427.438615] Lustre: server umount lustre-MDT0000 complete [ 3428.837705] LustreError: 101207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.851541] LustreError: 101207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3431.165670] LustreError: 99378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786692504 with bad export cookie 18105865007970013889 [ 3431.185467] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3431.185819] LustreError: 99378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3431.487218] Lustre: server umount lustre-MDT0001 complete [ 3443.718513] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3446.503235] Key type lgssc unregistered [ 3446.770155] LNet: 102955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.775208] LNetError: 102955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3446.791339] LNet: Removed LNI 192.168.202.106@tcp [ 3447.556387] Key type .llcrypt unregistered [ 3447.558632] Key type ._llcrypt unregistered [ 3466.198253] Key type ._llcrypt registered [ 3466.203399] Key type .llcrypt registered [ 3466.312705] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3467.334022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3467.449562] alg: No test for adler32 (adler32-zlib) [ 3468.433676] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 3468.733344] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3470.429253] Key type lgssc registered [ 3471.504348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3477.146148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3477.154502] LustreError: 104037:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3477.180597] LustreError: 104040:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3477.187456] LustreError: 104037:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4972 [ 3477.198099] LustreError: 104037:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3477.211742] LustreError: 104037:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3477.216980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.850212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3487.136551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.342980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3492.302196] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3498.062799] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lsmod [ 3501.272760] LustreError: 105237:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3501.325263] LustreError: 105245:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3501.331518] LustreError: 105237:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4946 [ 3501.341541] LustreError: 105237:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3501.351843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3501.365843] LustreError: 105237:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3501.835414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3503.956821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3508.192583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3508.199064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.224764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.218263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.222768] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.245231] Lustre: Skipped 1 previous similar message [ 3511.281815] Lustre: server umount lustre-MDT0000 complete [ 3512.356543] LustreError: 104077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.388841] LustreError: 104077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3515.295425] LustreError: 104054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786692588 with bad export cookie 14902034067293689187 [ 3515.303131] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3515.305936] LustreError: 104054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3516.511294] LustreError: 104508:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000a2786ab x1873482917892992/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3516.696140] Lustre: server umount lustre-MDT0001 complete [ 3531.239917] Lustre: server umount lustre-OST0000 complete [ 3541.031613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.401900] LustreError: 106310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3541.468272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.592446] LustreError: 106311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3547.208814] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3551.715918] LustreError: 106310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.855571] LustreError: 106311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3557.016676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.440712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3562.370625] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3571.778277] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3578.222961] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3585.159820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3585.639165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3586.679402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3591.719713] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3599.745395] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3613.672079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3613.680928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3613.683053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3613.683060] Lustre: Skipped 2 previous similar messages [ 3613.707965] Lustre: Skipped 1 previous similar message [ 3618.786574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3618.797143] Lustre: Skipped 1 previous similar message [ 3619.512947] Lustre: server umount lustre-OST0000 complete [ 3629.049416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.052639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3629.054465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3629.054472] Lustre: Skipped 1 previous similar message [ 3629.075125] Lustre: Skipped 1 previous similar message [ 3630.779473] Lustre: server umount lustre-MDT0000 complete [ 3634.145037] LustreError: 106305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3634.168496] LustreError: 106305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3634.665215] LustreError: 106291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786692707 with bad export cookie 14902034067293690118 [ 3634.669979] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3634.672181] LustreError: 106291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3635.092637] Lustre: server umount lustre-MDT0001 complete [ 3648.870532] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3652.139733] Key type lgssc unregistered [ 3652.545930] LNet: 109872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3652.550103] LNetError: 109872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3652.570208] LNet: Removed LNI 192.168.202.106@tcp [ 3653.762406] Key type .llcrypt unregistered [ 3653.764557] Key type ._llcrypt unregistered [ 3685.962363] Key type ._llcrypt registered [ 3685.969118] Key type .llcrypt registered [ 3686.142991] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3687.601841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3687.656716] alg: No test for adler32 (adler32-zlib) [ 3688.779193] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 3688.997757] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3690.669921] Key type lgssc registered [ 3692.271064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3702.001937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3702.027218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.584555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.554778] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3712.995073] LustreError: 111292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.014150] LustreError: 111292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3718.115230] LustreError: 111291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.677576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.162553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3724.820089] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3733.768240] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3740.048419] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3748.265609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3748.629903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3749.693150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3755.851451] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3765.195234] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3772.096738] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3777.341443] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3784.672272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3784.686527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3784.688815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3784.719838] Lustre: Skipped 1 previous similar message [ 3784.754546] Lustre: Skipped 1 previous similar message [ 3788.284270] Lustre: server umount lustre-OST0000 complete [ 3794.914941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.916924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3794.954140] Lustre: Skipped 1 previous similar message [ 3794.968958] Lustre: Skipped 1 previous similar message [ 3800.032536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.041123] Lustre: Skipped 1 previous similar message [ 3804.133825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3804.140487] Lustre: Skipped 1 previous similar message [ 3807.201795] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3807.361576] Lustre: server umount lustre-MDT0000 complete [ 3809.248939] LustreError: 111292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.268135] LustreError: 111292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3811.718169] LustreError: 111272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786692884 with bad export cookie 518884937119379883 [ 3811.732969] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.736978] LustreError: 111272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3812.397324] Lustre: server umount lustre-MDT0001 complete [ 3826.586866] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3829.286584] Key type lgssc unregistered [ 3829.646789] LNet: 114947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.653538] LNetError: 114947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3829.671330] LNet: Removed LNI 192.168.202.106@tcp [ 3830.464455] Key type .llcrypt unregistered [ 3830.467142] Key type ._llcrypt unregistered [ 3850.781597] Key type ._llcrypt registered [ 3850.784683] Key type .llcrypt registered [ 3850.889976] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 3851.984511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3852.114614] alg: No test for adler32 (adler32-zlib) [ 3853.255583] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 3853.572561] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 3855.311275] Key type lgssc registered [ 3856.637723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3864.854548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3864.895508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.694771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3871.960765] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3876.836890] LustreError: 116169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.856765] LustreError: 116169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3881.957509] LustreError: 116168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3883.238064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.899191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.882637] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3899.264571] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3906.159562] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3914.180214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3914.536806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.994774] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 3928.697993] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3936.062526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3949.551917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3949.563449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.582242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.588595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3951.596248] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.615775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3952.957601] Lustre: server umount lustre-OST0000 complete [ 3960.805827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3960.816987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.848252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.863170] Lustre: Skipped 1 previous similar message [ 3964.229648] Lustre: server umount lustre-MDT0000 complete [ 3966.949490] LustreError: 118660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.970932] LustreError: 118660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3968.616459] LustreError: 118013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786693041 with bad export cookie 6928258125811638032 [ 3968.623315] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.636826] LustreError: 118013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3969.329608] Lustre: server umount lustre-MDT0001 complete [ 3984.603580] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 3988.877672] Key type lgssc unregistered [ 3989.153375] LNet: 119731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3989.176289] LNetError: 119731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3989.196326] LNet: Removed LNI 192.168.202.106@tcp [ 3990.588591] Key type .llcrypt unregistered [ 3990.592650] Key type ._llcrypt unregistered [ 4018.414152] Key type ._llcrypt registered [ 4018.416215] Key type .llcrypt registered [ 4018.543819] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4019.747068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4019.787095] alg: No test for adler32 (adler32-zlib) [ 4021.080590] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 4021.487891] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4023.239480] Key type lgssc registered [ 4024.637791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4033.659792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4033.706487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.192391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.439537] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4045.823785] LustreError: 121149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.856281] LustreError: 121149:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4050.912659] LustreError: 121150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.104814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.495311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4055.799823] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4064.255309] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4070.488272] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4077.345497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4077.671616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.768251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 4084.260604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4093.523522] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.831924] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4110.512548] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4110.516953] Lustre: Skipped 2 previous similar messages [ 4112.556823] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4112.584584] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4115.335727] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4115.340582] Lustre: Skipped 2 previous similar messages [ 4118.319908] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4118.329150] Lustre: Skipped 1 previous similar message [ 4125.422055] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4125.430413] Lustre: Skipped 2 previous similar messages [ 4125.600349] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.6@tcp 192.168.0.[2,10]@tcp [ 4131.658290] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.6/24@tcp in log lustre-MDT0000 [ 4131.664309] Lustre: Skipped 2 previous similar messages [ 4131.737756] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.6/24@tcp [ 4131.748498] Lustre: Skipped 1 previous similar message [ 4141.745751] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 4143.601507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4143.612164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4143.616166] Lustre: Skipped 1 previous similar message [ 4143.623263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4143.629705] Lustre: Skipped 1 previous similar message [ 4148.710359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4148.723153] Lustre: Skipped 1 previous similar message [ 4149.386724] Lustre: server umount lustre-OST0000 complete [ 4153.831169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4153.835490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.841977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4153.864642] Lustre: Skipped 1 previous similar message [ 4158.946217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4158.956332] Lustre: Skipped 2 previous similar messages [ 4164.070959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4168.159141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4169.190830] LustreError: 122391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4169.211763] LustreError: 122391:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4170.512250] Lustre: server umount lustre-MDT0000 complete [ 4173.285278] LustreError: 121144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4173.309354] LustreError: 121144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4174.606198] LustreError: 121131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786693247 with bad export cookie 5274606156847072388 [ 4174.612181] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.614760] LustreError: 121131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4174.890600] Lustre: server umount lustre-MDT0001 complete [ 4187.633260] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4191.012960] Key type lgssc unregistered [ 4191.360557] LNet: 125811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4191.367806] LNetError: 125811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4192.423767] LNet: Removed LNI 192.168.202.106@tcp [ 4193.446278] Key type .llcrypt unregistered [ 4193.448676] Key type ._llcrypt unregistered [ 4226.226812] Key type ._llcrypt registered [ 4226.229833] Key type .llcrypt registered [ 4226.449758] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4228.198284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4228.259480] alg: No test for adler32 (adler32-zlib) [ 4229.355855] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 4229.671811] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4231.311203] Key type lgssc registered [ 4232.753756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4241.551890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4241.565521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4243.217107] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4243.246707] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.6/24@tcp [ 4243.247513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.740756] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4253.668845] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.683099] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4257.855802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.548376] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4258.582266] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.6/24@tcp [ 4258.656571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.137833] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4276.189051] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4284.993545] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4293.184127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4293.676901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4300.291842] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4308.482693] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4313.162245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4328.425833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.426419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4328.452086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4328.454812] Lustre: Skipped 1 previous similar message [ 4330.716652] Lustre: server umount lustre-OST0000 complete [ 4338.656809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.670686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4338.678638] Lustre: Skipped 1 previous similar message [ 4338.705743] Lustre: Skipped 2 previous similar messages [ 4342.267693] Lustre: server umount lustre-MDT0000 complete [ 4343.781636] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4343.808794] LustreError: 127375:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4347.091275] LustreError: 127356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786693420 with bad export cookie 18141757026962832929 [ 4347.115734] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4347.122382] LustreError: 127356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4347.716753] Lustre: server umount lustre-MDT0001 complete [ 4362.183870] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4364.638798] Key type lgssc unregistered [ 4364.921734] LNet: 130984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4364.951159] LNetError: 130984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4365.990710] LNet: Removed LNI 192.168.202.106@tcp [ 4367.189625] Key type .llcrypt unregistered [ 4367.191839] Key type ._llcrypt unregistered [ 4398.689749] Key type ._llcrypt registered [ 4398.692215] Key type .llcrypt registered [ 4398.789559] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4415.265631] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4416.039781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4416.057264] alg: No test for adler32 (adler32-zlib) [ 4417.041067] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 4417.285643] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4418.983189] Key type lgssc registered [ 4419.913356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4457.529599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4457.579489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4458.845726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4458.863373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4458.914980] Lustre: lustre-MDT0000: new disk, initializing [ 4458.972816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.988503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4466.142339] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4479.074706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.196826] Lustre: 134385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4479.251955] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4479.257139] Lustre: Skipped 1 previous similar message [ 4479.343596] Lustre: lustre-MDT0001: new disk, initializing [ 4479.474892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4479.528329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4479.539325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4483.977700] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4494.885832] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4501.865192] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4509.630247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4509.923321] Lustre: lustre-OST0000: new disk, initializing [ 4509.934762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4509.955520] Lustre: 136356:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4510.029300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4517.017845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4518.998384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4519.014363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4519.188569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4527.430662] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4532.700585] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4532.880754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4539.494429] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4539.921212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4545.008667] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4545.016992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.023652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4545.032870] Lustre: Skipped 1 previous similar message [ 4550.124313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.134633] Lustre: Skipped 2 previous similar messages [ 4555.245607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.258575] Lustre: Skipped 1 previous similar message [ 4556.255409] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4556.446369] Lustre: server umount lustre-OST0000 complete [ 4560.361748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.371896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4560.388372] Lustre: Skipped 1 previous similar message [ 4560.404401] Lustre: Skipped 1 previous similar message [ 4565.472133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4565.479429] Lustre: Skipped 1 previous similar message [ 4574.176791] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4574.400435] Lustre: server umount lustre-MDT0000 complete [ 4578.271235] LustreError: 136355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786693651 with bad export cookie 3137172238282909678 [ 4578.280677] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4578.301403] LustreError: 136355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4578.905455] Lustre: server umount lustre-MDT0001 complete [ 4589.026392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4589.547858] LustreError: 138575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4589.560909] LustreError: 138575:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4589.648155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.027317] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4594.660995] LustreError: 138576:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.794152] LustreError: 138575:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4603.645571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.993890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4608.644392] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4617.492863] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4625.418051] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4632.570971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.934069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.530114] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4647.902990] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4652.946104] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4653.316083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4658.647570] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4658.934965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4668.142963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4668.360989] Lustre: lustre-OST0001: new disk, initializing [ 4668.365677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4668.372519] Lustre: 141950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4668.456061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4674.955818] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4676.713596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4676.723860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4676.807200] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4683.517790] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4683.733751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4689.633371] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4689.900226] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.224552] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d0708522800.ost_server_uuid 50 [ 4699.607526] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d0708522800.ost_server_uuid in IDLE state after 0 sec [ 4712.931125] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4712.938358] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.954585] LustreError: Skipped 2 previous similar messages [ 4712.957951] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4712.957958] Lustre: Skipped 2 previous similar messages [ 4713.009732] Lustre: Skipped 1 previous similar message [ 4724.192502] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4724.337210] Lustre: server umount lustre-OST0001 complete [ 4728.299673] LustreError: 140400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4728.331305] LustreError: 140400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4730.336412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4730.341152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4730.357381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4730.370985] Lustre: Skipped 5 previous similar messages [ 4733.419719] LustreError: 141716:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.444694] LustreError: 141716:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4734.852965] Lustre: server umount lustre-OST0000 complete [ 4742.114971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4742.128604] LustreError: Skipped 1 previous similar message [ 4742.144333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.160272] Lustre: Skipped 1 previous similar message [ 4745.798705] Lustre: server umount lustre-MDT0000 complete [ 4748.768437] LustreError: 139304:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.822290] LustreError: 139304:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4751.105389] LustreError: 138557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786693824 with bad export cookie 3137172238282910560 [ 4751.137889] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4751.732182] Lustre: server umount lustre-MDT0001 complete [ 4775.524799] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 4778.995804] Key type lgssc unregistered [ 4779.319791] LNet: 144957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4779.346727] LNetError: 144957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4779.363399] LNet: Removed LNI 192.168.202.106@tcp [ 4780.456648] Key type .llcrypt unregistered [ 4780.462750] Key type ._llcrypt unregistered [ 4824.755608] Key type ._llcrypt registered [ 4824.759427] Key type .llcrypt registered [ 4824.875923] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 4842.094719] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 4844.039688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4844.138226] alg: No test for adler32 (adler32-zlib) [ 4845.389129] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 4845.868852] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 4847.535867] Key type lgssc registered [ 4848.830411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4892.107171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4892.155717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4893.561638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4893.677308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4893.775785] Lustre: lustre-MDT0000: new disk, initializing [ 4893.880932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.911961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4899.761491] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4911.286290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.410554] Lustre: 149121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4911.467669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4911.476611] Lustre: Skipped 1 previous similar message [ 4911.585786] Lustre: lustre-MDT0001: new disk, initializing [ 4911.719316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4911.752864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4911.768059] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4916.770379] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4928.787523] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4935.955328] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4943.205679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4943.572920] Lustre: lustre-OST0000: new disk, initializing [ 4943.582868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4943.595576] Lustre: 151091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4943.709788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4950.114898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4950.128557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4950.276968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4950.660789] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4961.210044] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4966.596234] Lustre: Failing over lustre-OST0000 [ 4966.800794] Lustre: server umount lustre-OST0000 complete [ 4984.290983] Lustre: 147344:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786694041/real 1786694042] req@00000000e5ad33fc x1873484361796224/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1786694057 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4984.337316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.513372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.905941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.958656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4986.974258] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4987.344394] Lustre: 147342:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786694044/real 1786694044] req@00000000ed596584 x1873484361797376/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786694060 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4989.987994] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4992.035475] Lustre: 147343:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786694049/real 1786694049] req@000000007acb4bb8 x1873484361798400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786694065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4992.077919] Lustre: 147343:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4993.808465] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 4995.685237] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4995.687522] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4997.635288] Lustre: Failing over lustre-MDT0000 [ 4997.801968] Lustre: server umount lustre-MDT0000 complete [ 4998.634145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4998.647807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5000.159152] Lustre: 147343:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786694057/real 1786694057] req@000000006780f681 x1873484361800576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786694073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5000.185479] Lustre: 147343:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5004.257390] LustreError: 149132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5004.284307] LustreError: 149132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5005.917267] LustreError: 149129:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.378309] LustreError: 149126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.397160] LustreError: 149126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5014.508778] LustreError: 151345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5014.545862] LustreError: 151345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5015.519431] Lustre: 147343:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786694072/real 1786694072] req@00000000e7eeba0a x1873484361810560/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786694088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5015.566514] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5019.831930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.643368] LustreError: 149133:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5020.681161] LustreError: 149133:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5025.807753] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcdce246cd151a7cb [ 5025.826450] Lustre: MGC192.168.202.106@tcp: Connection restored to 0@lo (at 0@lo) [ 5025.839636] Lustre: Skipped 1 previous similar message [ 5026.087258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5026.142205] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5027.413603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5031.244441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5031.401336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5031.420262] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5031.466956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5041.642980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.649910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5041.659436] Lustre: Skipped 3 previous similar messages [ 5041.685134] Lustre: Skipped 1 previous similar message [ 5046.589086] Lustre: server umount lustre-OST0000 complete [ 5051.874917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.885827] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5051.909245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.006613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5057.016145] Lustre: Skipped 2 previous similar messages [ 5057.331456] Lustre: server umount lustre-MDT0000 complete [ 5061.092473] LustreError: 149127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.118423] LustreError: 149127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5061.916791] LustreError: 149111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786694134 with bad export cookie 14829830672764479435 [ 5061.918531] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5061.926479] LustreError: 149111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5062.301919] Lustre: server umount lustre-MDT0001 complete [ 5074.321041] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 5077.235923] Key type lgssc unregistered [ 5077.520613] LNet: 154753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5077.535195] LNetError: 154753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5077.555974] LNet: Removed LNI 192.168.202.106@tcp [ 5078.520558] Key type .llcrypt unregistered [ 5078.524576] Key type ._llcrypt unregistered [ 5120.183962] Key type ._llcrypt registered [ 5120.187722] Key type .llcrypt registered [ 5120.276551] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5121.509699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5121.523263] alg: No test for adler32 (adler32-zlib) [ 5122.656302] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 5122.906204] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 5124.615200] Key type lgssc registered [ 5126.172673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5135.377300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5135.417197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.872667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.515334] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5146.086140] LustreError: 156278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.107274] LustreError: 156278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5151.209680] LustreError: 156277:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.988305] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5156.341939] LustreError: 156278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.234919] LustreError: 156306:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5157.249680] LustreError: 156306:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5157.263526] LustreError: 156306:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5158.030854] Lustre: server umount lustre-MDT0000 complete [ 5176.517926] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5188.989156] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5224.437748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.646491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5224.675762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5224.741324] Lustre: lustre-MDT0000: new disk, initializing [ 5224.837775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.862786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5228.879655] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5241.198540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.470909] Lustre: 159613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5241.576061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5241.580334] Lustre: Skipped 1 previous similar message [ 5241.732605] Lustre: lustre-MDT0001: new disk, initializing [ 5241.991288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.088988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5242.117669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5250.108266] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5261.440602] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.481194] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5275.669943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.945433] Lustre: lustre-OST0000: new disk, initializing [ 5275.948619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5275.956317] Lustre: 161585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5276.034273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.749983] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5284.954222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5284.966776] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5285.098149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5291.854613] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5295.867238] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5297.922968] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:59:29 (1786694369) [ 5566.945548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.960842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5566.976938] Lustre: Skipped 2 previous similar messages [ 5571.855624] Lustre: server umount lustre-MDT0000 complete [ 5572.079763] LustreError: 160985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5572.122437] LustreError: 160985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5576.322268] LustreError: 159607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786694649 with bad export cookie 10538027039690995244 [ 5576.333536] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5577.183982] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5577.184178] LustreError: 159622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5577.206534] Lustre: Skipped 2 previous similar messages [ 5577.207803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5581.217780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5582.771524] Lustre: server umount lustre-MDT0001 complete [ 5593.418405] Lustre: server umount lustre-OST0000 complete [ 5604.144995] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5616.257244] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5660.265411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5660.698651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5660.743339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5660.887205] Lustre: lustre-MDT0000: new disk, initializing [ 5661.040487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5661.077122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5666.894962] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5680.739600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.928104] Lustre: 166266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5681.022691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5681.027393] Lustre: Skipped 1 previous similar message [ 5681.166757] Lustre: lustre-MDT0001: new disk, initializing [ 5681.314101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5681.364531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5681.384475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5686.882819] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5697.750144] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5705.788759] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5714.143166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.715288] Lustre: lustre-OST0000: new disk, initializing [ 5714.727967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5714.747989] Lustre: 168234:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5714.887204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5723.584919] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5724.259976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5724.273050] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5724.412626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5735.192902] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5740.705724] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5740.942595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5746.450882] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5746.732542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5750.260246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5750.264919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5750.273503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5750.283103] Lustre: Skipped 1 previous similar message [ 5750.315818] Lustre: Skipped 1 previous similar message [ 5754.958485] Lustre: server umount lustre-OST0000 complete [ 5760.485178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.499128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.505028] Lustre: Skipped 1 previous similar message [ 5765.619700] LustreError: 166276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.636971] LustreError: 166276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5765.912418] Lustre: server umount lustre-MDT0000 complete [ 5770.404382] LustreError: 166258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786694843 with bad export cookie 10538027039691154067 [ 5770.424101] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5770.425123] LustreError: 166258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5771.016123] Lustre: server umount lustre-MDT0001 complete [ 5782.097809] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:07:33 (1786694853) [ 5802.818421] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5815.128964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 5854.759955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.158509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.196394] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.331886] Lustre: lustre-MDT0000: new disk, initializing [ 5855.433078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.463253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5860.245615] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5872.876826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.035694] Lustre: 172992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5873.235885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5879.876479] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5892.816970] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5899.330031] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5906.396089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.683922] Lustre: lustre-OST0000: new disk, initializing [ 5906.686936] Lustre: Skipped 1 previous similar message [ 5906.691756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5906.697609] Lustre: Skipped 2 previous similar messages [ 5906.708814] Lustre: 174969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5914.751240] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 5914.768084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5914.774343] Lustre: Skipped 1 previous similar message [ 5914.787809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5914.942762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5925.916665] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5935.584569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5935.596491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5935.603518] LustreError: Skipped 2 previous similar messages [ 5935.619578] Lustre: Skipped 1 previous similar message [ 5935.625337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5935.643846] Lustre: Skipped 1 previous similar message [ 5940.190218] Lustre: server umount lustre-OST0000 complete [ 5944.800425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5944.816269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.828885] Lustre: Skipped 1 previous similar message [ 5950.910352] Lustre: server umount lustre-MDT0000 complete [ 5950.946772] LustreError: 173003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5950.967748] LustreError: 173003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5954.745667] LustreError: 172984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695027 with bad export cookie 10538027039691154949 [ 5954.749948] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5954.768302] LustreError: 172984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5955.080723] Lustre: server umount lustre-MDT0001 complete [ 5965.471272] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:10:36 (1786695036) [ 5986.797555] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 5999.888107] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6046.467068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6046.964738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6047.010600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6047.205792] Lustre: lustre-MDT0000: new disk, initializing [ 6047.323103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.327266] Lustre: Skipped 2 previous similar messages [ 6047.344268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6052.676918] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6063.856486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6064.013485] Lustre: 179299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6064.298630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6069.000363] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6079.442249] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6086.111670] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.831887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.141695] Lustre: 181275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6100.495285] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6101.558417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6101.673799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6110.269828] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6121.957151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.972511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.984402] Lustre: Skipped 1 previous similar message [ 6121.992572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.005413] Lustre: Skipped 3 previous similar messages [ 6124.402781] Lustre: server umount lustre-OST0000 complete [ 6131.173165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6131.185449] LustreError: Skipped 1 previous similar message [ 6134.452744] Lustre: server umount lustre-MDT0000 complete [ 6137.834863] LustreError: 180116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.859017] LustreError: 180116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6139.096803] LustreError: 181274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695212 with bad export cookie 10538027039691156041 [ 6139.110661] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.117209] LustreError: 181274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6139.720994] Lustre: server umount lustre-MDT0001 complete [ 6152.209821] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6156.204960] Key type lgssc unregistered [ 6156.608065] LNet: 183132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6156.631204] LNetError: 183132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6156.649987] LNet: Removed LNI 192.168.202.106@tcp [ 6157.852965] Key type .llcrypt unregistered [ 6157.857799] Key type ._llcrypt unregistered [ 6182.636656] Key type ._llcrypt registered [ 6182.638925] Key type .llcrypt registered [ 6182.757581] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6183.921023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6183.973287] alg: No test for adler32 (adler32-zlib) [ 6185.046680] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 6185.281492] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6186.935188] Key type lgssc registered [ 6188.205403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.360849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6196.383817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.884394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.293432] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6207.467488] LustreError: 184551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6207.491370] LustreError: 184551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6212.095976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.543737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6217.874304] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6226.505502] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.925394] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6240.675548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.233604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6242.234937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6248.757701] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6258.480596] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.123192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.126470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.133239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.149572] Lustre: Skipped 1 previous similar message [ 6283.250178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.258947] Lustre: Skipped 2 previous similar messages [ 6284.096224] Lustre: server umount lustre-OST0000 complete [ 6288.356141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.374374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6288.383475] Lustre: Skipped 1 previous similar message [ 6293.474194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6293.483494] Lustre: Skipped 1 previous similar message [ 6294.807130] Lustre: server umount lustre-MDT0000 complete [ 6298.600085] LustreError: 184547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.630520] LustreError: 184547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6299.651180] LustreError: 184532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695372 with bad export cookie 289423206364792153 [ 6299.668744] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.972885] Lustre: server umount lustre-MDT0001 complete [ 6314.011941] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 6317.037556] Key type lgssc unregistered [ 6317.416847] LNet: 188115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6317.421271] LNetError: 188115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6317.433342] LNet: Removed LNI 192.168.202.106@tcp [ 6318.346706] Key type .llcrypt unregistered [ 6318.348710] Key type ._llcrypt unregistered [ 6345.795361] Key type ._llcrypt registered [ 6345.799034] Key type .llcrypt registered [ 6345.960400] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6347.145694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6347.234164] alg: No test for adler32 (adler32-zlib) [ 6348.431609] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 6348.755942] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 6350.447168] Key type lgssc registered [ 6351.763953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6361.009245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6361.031313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6362.816321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.435160] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6372.330817] LustreError: 189535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6372.367284] LustreError: 189535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6377.444715] LustreError: 189536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6379.965805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6380.529707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6386.559527] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6394.453595] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6401.147205] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6409.084671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.469788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6415.742379] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6424.366780] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6431.992612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6441.443279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6441.458650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6441.487778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.473376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6442.474360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6442.483480] Lustre: Skipped 1 previous similar message [ 6442.499793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.662155] Lustre: lustre-OST0000: Not available for connect from 192.168.202.6@tcp (stopping) [ 6446.672939] Lustre: Skipped 1 previous similar message [ 6447.050761] Lustre: server umount lustre-OST0000 complete [ 6455.745389] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d0705b8b000.ost_server_uuid 50 [ 6457.289883] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6471.124760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.133840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6471.144853] Lustre: Skipped 1 previous similar message [ 6477.282230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6477.287768] Lustre: Skipped 4 previous similar messages [ 6479.430495] Lustre: server umount lustre-MDT0000 complete [ 6482.400667] LustreError: 189536:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.440396] LustreError: 189536:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6483.295368] LustreError: 192135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695556 with bad export cookie 11243329272779973042 [ 6483.296234] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.301697] LustreError: 192135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6483.695721] Lustre: server umount lustre-MDT0001 complete [ 6493.381704] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:19:25 (1786695565) [ 6500.217701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.595126] LustreError: 193478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6500.659610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.302788] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6505.956732] LustreError: 193479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6513.706992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.000433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6518.141789] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6527.767157] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6533.186131] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6541.752020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6542.163811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6545.284430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6550.420405] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6559.457769] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.223240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6567.455854] Lustre: lustre-OST0001: new disk, initializing [ 6567.465052] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6567.481739] Lustre: 196318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6567.590293] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.718333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6573.733945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6573.841920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6574.067206] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6583.969365] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6589.420033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6589.438773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6589.453190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.466320] Lustre: Skipped 1 previous similar message [ 6602.207702] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6602.369884] Lustre: server umount lustre-OST0000 complete [ 6604.795175] LustreError: 195300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6604.845902] LustreError: 195300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6612.025708] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6612.407989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6617.913346] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6618.249831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6625.250643] LustreError: 195300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6625.276520] LustreError: 195300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6635.492532] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6635.499024] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6635.509370] Lustre: Skipped 7 previous similar messages [ 6635.520352] Lustre: Skipped 2 previous similar messages [ 6638.786493] Lustre: server umount lustre-OST0001 complete [ 6642.150307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6645.733178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.743654] Lustre: Skipped 1 previous similar message [ 6648.446189] Lustre: server umount lustre-MDT0000 complete [ 6652.266061] LustreError: 193459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695725 with bad export cookie 11243329272779974113 [ 6652.268237] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6652.273347] LustreError: 193459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6652.526377] Lustre: server umount lustre-MDT0001 complete [ 6675.901265] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:22:27 (1786695747) [ 6684.263734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6684.741411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6684.818509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6685.269288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6692.528235] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6703.233359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.402902] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6703.463844] Lustre: 199758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6709.213819] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6717.508447] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6724.131635] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6731.304987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6731.523239] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6738.722611] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6739.047897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6747.216959] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6756.027415] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6756.101498] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6756.346851] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6756.351574] Lustre: Skipped 2 previous similar messages [ 6762.980946] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6771.229991] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6774.754207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6774.758486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.766494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6774.769324] Lustre: Skipped 4 previous similar messages [ 6780.742776] Lustre: server umount lustre-OST0000 complete [ 6781.419795] LustreError: 201609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6781.437781] LustreError: 201609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 6796.772242] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6796.783408] LustreError: Skipped 1 previous similar message [ 6802.530563] Lustre: server umount lustre-OST0001 complete [ 6807.011753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.025112] Lustre: Skipped 3 previous similar messages [ 6821.343570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6821.954897] Lustre: server umount lustre-MDT0000 complete [ 6826.119963] LustreError: 199751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786695899 with bad export cookie 11243329272779975513 [ 6826.121357] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.133330] LustreError: 199751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6826.533972] Lustre: server umount lustre-MDT0001 complete [ 6853.621644] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:25:25 (1786695925) [ 6871.460433] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 6885.937856] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 6926.241930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.691283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6926.726444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6926.807760] Lustre: lustre-MDT0000: new disk, initializing [ 6926.903101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6926.923941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6932.124521] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6943.135969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.291893] Lustre: 207857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6943.422258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6943.435856] Lustre: Skipped 1 previous similar message [ 6943.591227] Lustre: lustre-MDT0001: new disk, initializing [ 6943.765535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6943.792434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6948.720525] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6959.162588] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6965.916596] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6973.810427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6974.233258] Lustre: lustre-OST0000: new disk, initializing [ 6974.251857] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6974.263125] Lustre: 209825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6981.185781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6981.204406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6981.316543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6981.620181] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 6992.171616] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6997.854150] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6998.093449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7003.384655] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7003.635429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7007.203213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7007.210298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7007.213574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7007.213588] Lustre: Skipped 14 previous similar messages [ 7007.221526] Lustre: Skipped 2 previous similar messages [ 7012.038123] Lustre: server umount lustre-OST0000 complete [ 7022.411485] Lustre: server umount lustre-MDT0000 complete [ 7022.567571] LustreError: 210290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.602821] LustreError: 210290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 7027.061857] LustreError: 207849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786696100 with bad export cookie 11243329272779977081 [ 7027.076823] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7027.084261] LustreError: 207849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7027.522965] Lustre: server umount lustre-MDT0001 complete [ 7037.081040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7041.892201] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7051.020277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.208485] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7064.100083] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7069.767893] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7077.409316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.684889] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7099.445910] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7104.657409] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7104.895479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7110.101423] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7110.469721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7116.773591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7119.398871] Lustre: server umount lustre-OST0000 complete [ 7128.912434] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7134.210426] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7163.462945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7172.139093] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7181.357513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7181.368926] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7181.370275] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7182.219564] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.374269] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7191.531041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.537923] Lustre: Skipped 4 previous similar messages [ 7191.538670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7191.541237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7191.541244] Lustre: Skipped 5 previous similar messages [ 7196.109365] Lustre: server umount lustre-OST0000 complete [ 7206.298261] Lustre: server umount lustre-MDT0000 complete [ 7206.884305] LustreError: 212038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.910456] LustreError: 212038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7210.115569] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786696283 with bad export cookie 11243329272779977963 [ 7210.122332] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.127765] LustreError: 212023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7210.486362] Lustre: server umount lustre-MDT0001 complete [ 7219.067910] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:31:30 (1786696290) [ 7225.853304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.597101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7226.607409] Lustre: Skipped 6 previous similar messages [ 7231.691985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7243.003464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7249.647091] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7258.701235] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7264.375234] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7270.858153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7277.235965] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7284.997601] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7290.744869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7291.021066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7296.572845] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7296.880412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7305.069992] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7305.334333] Lustre: lustre-OST0001: new disk, initializing [ 7305.340352] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7305.355525] Lustre: 221390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7313.660266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7313.673181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7313.889665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7313.939237] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7325.449544] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7330.692934] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7330.878894] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7334.962167] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7335.201939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7339.512690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7339.525745] LustreError: Skipped 2 previous similar messages [ 7343.415254] Lustre: server umount lustre-OST0001 complete [ 7349.072526] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7349.240569] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7353.772033] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7353.999493] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7383.620678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.170921] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7389.208741] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7389.212158] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7389.244892] Lustre: Skipped 2 previous similar messages [ 7391.893607] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7402.638996] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.630725] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7414.241278] Lustre: server umount lustre-OST0001 complete [ 7437.260147] Lustre: server umount lustre-OST0000 complete [ 7448.723023] Lustre: server umount lustre-MDT0000 complete [ 7452.845260] LustreError: 218041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786696525 with bad export cookie 11243329272779978922 [ 7452.852173] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.857681] LustreError: 218041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7459.373664] Lustre: server umount lustre-MDT0001 complete [ 7487.167104] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:35:58 (1786696558) [ 7494.740921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.004836] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7495.019900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7500.176733] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7510.069586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.336647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7510.392587] Lustre: 226631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7516.520290] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7524.926783] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7532.658298] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7541.273033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.568230] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7549.099737] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7558.357403] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7569.469274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.539143] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7577.627271] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7586.334633] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7587.974936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7593.973482] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7594.148398] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7598.899968] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7599.195110] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7605.457826] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d07116b3000.ost_server_uuid 50 [ 7607.315895] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d07116b3000.ost_server_uuid in IDLE state after 0 sec [ 7608.574669] Lustre: Permanently deactivating lustre-OST0001 [ 7608.598848] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7612.286778] Lustre: Permanently reactivating lustre-OST0001 [ 7619.049923] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7619.057255] LustreError: Skipped 5 previous similar messages [ 7619.060928] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7619.072138] Lustre: Skipped 11 previous similar messages [ 7619.077374] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7619.081937] Lustre: Skipped 15 previous similar messages [ 7623.172916] Lustre: server umount lustre-OST0001 complete [ 7624.679357] LustreError: 229895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.697849] LustreError: 229895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 7633.120361] Lustre: server umount lustre-OST0000 complete [ 7644.156321] Lustre: server umount lustre-MDT0000 complete [ 7648.241985] LustreError: 227374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786696721 with bad export cookie 11243329272779980210 [ 7648.249271] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7648.251483] LustreError: 227374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7648.638835] Lustre: server umount lustre-MDT0001 complete [ 7674.414460] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:39:05 (1786696745) [ 7681.882449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.212449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7682.276616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7682.287242] Lustre: Skipped 5 previous similar messages [ 7686.583892] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7694.135506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.232125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7694.260720] Lustre: 233326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7699.488651] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7707.452485] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7713.639621] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7721.205783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7721.333202] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7721.346684] Lustre: Permanently deactivating lustre-OST0000 [ 7721.356244] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7727.155329] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7729.247045] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7736.816404] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.540067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.623664] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7742.834290] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7742.840948] Lustre: Skipped 11 previous similar messages [ 7749.021288] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7749.825667] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7756.547499] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7758.373614] Lustre: Permanently reactivating lustre-OST0000 [ 7758.398450] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7758.403317] Lustre: Skipped 2 previous similar messages [ 7767.689570] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7767.698272] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 000000004cdbf1e3, cur 1786696840 deadline 1786696830 last 1786696800 [ 7767.704074] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7767.739477] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7767.798754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7779.092049] Lustre: server umount lustre-OST0001 complete [ 7789.462455] Lustre: server umount lustre-OST0000 complete [ 7800.375187] Lustre: server umount lustre-MDT0000 complete [ 7804.510420] LustreError: 233319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786696877 with bad export cookie 11243329272779981862 [ 7804.517220] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.525963] LustreError: 233319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7805.230643] Lustre: server umount lustre-MDT0001 complete [ 7817.377687] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 7820.339520] Key type lgssc unregistered [ 7820.605788] LNet: 238208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.614164] LNetError: 238208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.627851] LNet: Removed LNI 192.168.202.106@tcp [ 7821.692672] Key type .llcrypt unregistered [ 7821.694966] Key type ._llcrypt unregistered [ 7843.768673] Key type ._llcrypt registered [ 7843.771470] Key type .llcrypt registered [ 7843.898626] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 7845.155652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7845.337869] alg: No test for adler32 (adler32-zlib) [ 7846.421515] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 7846.621711] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 7848.271197] Key type lgssc registered [ 7850.244922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7862.857027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7862.891146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.346589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7868.723763] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7874.535408] LustreError: 239657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.549977] LustreError: 239657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7877.769425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.892448] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7877.924921] Lustre: 239644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7877.934917] Lustre: Permanently deactivating lustre-MDT0001 [ 7877.955185] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7878.288035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7880.012320] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7880.031724] LustreError: 239687:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7880.047101] LustreError: 239687:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7880.063416] LustreError: 239687:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7882.553793] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7890.068635] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7895.598265] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7902.749670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.073558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.126759] LustreError: 241497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.224688] LustreError: 241497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.245195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7909.263657] LustreError: 241497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7910.080786] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7914.473147] LustreError: 241495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7914.498837] LustreError: 241495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7918.459199] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7919.599132] LustreError: 241495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7919.615085] LustreError: 241495:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7925.509380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7925.680206] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.897697] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7930.912664] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7932.559732] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 7942.558335] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7947.363212] Lustre: Permanently reactivating lustre-MDT0001 [ 7947.416958] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7947.420870] Lustre: Skipped 2 previous similar messages [ 7951.468539] Lustre: lustre-MDT0001: Client bce014be-52a4-4f52-b3de-3a99954c234b (at 192.168.202.6@tcp) reconnecting [ 7958.446469] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7959.823988] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7965.656969] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7965.903267] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7971.021517] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7971.438595] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7975.006777] Lustre: Permanently deactivating lustre-MDT0001 [ 7975.018514] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7975.022724] Lustre: Skipped 2 previous similar messages [ 7984.810305] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7992.299896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.307976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7992.318312] Lustre: Skipped 3 previous similar messages [ 7992.319508] Lustre: Skipped 2 previous similar messages [ 7997.411352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7997.420214] Lustre: Skipped 4 previous similar messages [ 7998.452964] Lustre: server umount lustre-MDT0000 complete [ 8002.412580] LustreError: 243205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786697075 with bad export cookie 325055275685284667 [ 8002.431727] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.532212] LustreError: 240916:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8002.556332] LustreError: 240916:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8002.764960] Lustre: server umount lustre-MDT0001 complete [ 8009.053670] Lustre: server umount lustre-OST0000 complete [ 8020.065560] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:44:51 (1786697091) [ 8031.992971] Lustre: server umount lustre-OST0001 complete [ 8039.535493] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 8048.772239] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8087.997589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8088.445602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8088.454634] Lustre: Skipped 2 previous similar messages [ 8088.487312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8088.561195] Lustre: lustre-MDT0000: new disk, initializing [ 8088.631602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8088.647523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8094.383681] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8106.144943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8106.274924] Lustre: 247939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8106.366822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8106.379501] Lustre: Skipped 1 previous similar message [ 8106.393748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8106.463820] Lustre: lustre-MDT0001: new disk, initializing [ 8106.540532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8106.581192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8106.602200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8110.793710] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8122.073706] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8130.286777] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8138.360074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8138.796471] Lustre: lustre-OST0000: new disk, initializing [ 8138.803662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8138.817741] Lustre: 249908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8138.918409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.869878] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8147.029834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8147.037632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8147.127797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8156.281573] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8159.125433] LustreError: 247944:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8167.439035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.612575] Lustre: lustre-OST0001: new disk, initializing [ 8167.616379] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8167.621527] Lustre: 251023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8169.185441] LustreError: 247944:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8169.210805] LustreError: 250500:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8173.693579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8173.727102] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8173.898142] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8175.761587] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8179.223125] LustreError: 250500:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8188.282921] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8190.781524] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8194.547921] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8194.548796] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.553688] LustreError: Skipped 1 previous similar message [ 8194.554810] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8194.573671] Lustre: Skipped 1 previous similar message [ 8199.664695] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8199.674509] Lustre: Skipped 2 previous similar messages [ 8200.734903] Lustre: server umount lustre-OST0001 complete [ 8204.783899] LustreError: 250333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8204.794038] LustreError: 250333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8224.224918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8224.242921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8224.254396] LustreError: Skipped 1 previous similar message [ 8224.264015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8224.275859] Lustre: Skipped 1 previous similar message [ 8224.304415] Lustre: Skipped 1 previous similar message [ 8228.508464] Lustre: server umount lustre-OST0000 complete [ 8234.464682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8234.474547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.734698] Lustre: server umount lustre-MDT0000 complete [ 8240.610756] LustreError: 249042:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8240.633749] LustreError: 249042:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8244.995905] LustreError: 247929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786697318 with bad export cookie 325055275685287264 [ 8245.000937] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8245.007828] LustreError: 247929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8251.459672] Lustre: server umount lustre-MDT0001 complete [ 8263.301617] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8266.339314] Key type lgssc unregistered [ 8266.806702] LNet: 253066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8266.819798] LNetError: 253066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8266.844299] LNet: Removed LNI 192.168.202.106@tcp [ 8267.885235] Key type .llcrypt unregistered [ 8267.886805] Key type ._llcrypt unregistered [ 8311.234730] Key type ._llcrypt registered [ 8311.236829] Key type .llcrypt registered [ 8311.320191] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8312.759680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8312.786071] alg: No test for adler32 (adler32-zlib) [ 8313.959416] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 8314.359902] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8316.064916] Key type lgssc registered [ 8317.634379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8326.607834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8326.620458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8328.020653] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8328.065603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8328.253335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.244257] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8344.046194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.139652] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8344.165498] Lustre: 255279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.440506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8349.525328] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8358.090367] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8365.904495] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8375.004675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8375.258528] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8375.611231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8382.589747] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8385.666473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8390.434537] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8446.946660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8446.955450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8446.977736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.463089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.484563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.488857] Lustre: Skipped 1 previous similar message [ 8451.316736] Lustre: server umount lustre-OST0000 complete [ 8456.137435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8469.502854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8469.927541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.994066] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8472.015371] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8477.492171] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8487.087770] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8488.440626] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8488.487685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8528.356721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8528.364077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.389100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8529.391825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8533.697761] Lustre: server umount lustre-OST0000 complete [ 8539.105376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8539.111132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.122367] Lustre: Skipped 1 previous similar message [ 8539.134416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8539.139571] Lustre: Skipped 2 previous similar messages [ 8544.759039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8544.776034] Lustre: Skipped 2 previous similar messages [ 8545.382986] Lustre: server umount lustre-MDT0000 complete [ 8551.149207] LustreError: 255273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786697624 with bad export cookie 16192479262001374225 [ 8551.157042] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8551.169226] LustreError: 255273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8552.093646] Lustre: server umount lustre-MDT0001 complete [ 8571.358435] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8575.275401] Key type lgssc unregistered [ 8575.603981] LNet: 260729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8575.621635] LNetError: 260729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8575.647827] LNet: Removed LNI 192.168.202.106@tcp [ 8576.812302] Key type .llcrypt unregistered [ 8576.817241] Key type ._llcrypt unregistered [ 8608.367205] Key type ._llcrypt registered [ 8608.370156] Key type .llcrypt registered [ 8608.466343] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8609.945308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8609.983876] alg: No test for adler32 (adler32-zlib) [ 8611.353980] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 8611.745934] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8613.488314] Key type lgssc registered [ 8615.915835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8626.104428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8626.151937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8627.692916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.730328] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8637.932897] LustreError: 262150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.967758] LustreError: 262150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8643.043663] LustreError: 262151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.739588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8644.473745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8650.046065] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8661.448958] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8670.537729] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8679.495212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.288567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8688.065925] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8697.391239] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8698.819711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8729.588124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.590978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8729.599224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.611382] Lustre: Skipped 1 previous similar message [ 8733.804841] Lustre: server umount lustre-OST0000 complete [ 8739.814713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.821989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8739.841235] Lustre: Skipped 1 previous similar message [ 8739.855432] Lustre: Skipped 2 previous similar messages [ 8744.541649] Lustre: server umount lustre-MDT0000 complete [ 8744.928604] LustreError: 262878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8744.956643] LustreError: 262878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8749.029939] LustreError: 262151:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.062627] LustreError: 262151:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8749.188655] LustreError: 262887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786697822 with bad export cookie 7547534756647152020 [ 8749.194922] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.207383] LustreError: 262887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8749.718335] Lustre: server umount lustre-MDT0001 complete [ 8764.333487] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8767.619111] Key type lgssc unregistered [ 8767.955269] LNet: 266460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8767.966237] LNetError: 266460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8767.988193] LNet: Removed LNI 192.168.202.106@tcp [ 8769.095839] Key type .llcrypt unregistered [ 8769.097775] Key type ._llcrypt unregistered [ 8788.257636] Key type ._llcrypt registered [ 8788.262414] Key type .llcrypt registered [ 8788.386015] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8789.249592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8789.274596] alg: No test for adler32 (adler32-zlib) [ 8790.386684] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 8790.603746] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8792.279374] Key type lgssc registered [ 8794.264994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8802.895254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8802.917047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.527310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.675451] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8814.052296] LustreError: 267681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8814.063733] LustreError: 267681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8819.177994] LustreError: 267682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8819.693839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8820.147778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8825.843377] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8834.397668] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8840.926466] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8850.058804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8850.742995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8859.018069] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8867.612476] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8874.303054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8894.957033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.961301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.963600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8894.971493] Lustre: Skipped 1 previous similar message [ 8897.653749] Lustre: server umount lustre-OST0000 complete [ 8905.186092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.191522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8905.209179] Lustre: Skipped 1 previous similar message [ 8905.232905] Lustre: Skipped 2 previous similar messages [ 8908.808348] Lustre: server umount lustre-MDT0000 complete [ 8910.318163] LustreError: 267681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8910.341095] LustreError: 267681:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8913.399929] LustreError: 267661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786697986 with bad export cookie 16274221893696410723 [ 8913.400542] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8913.414401] LustreError: 267661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8913.791668] Lustre: server umount lustre-MDT0001 complete [ 8925.966118] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 8928.843281] Key type lgssc unregistered [ 8929.170656] LNet: 271390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8929.183888] LNetError: 271390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8929.209476] LNet: Removed LNI 192.168.202.106@tcp [ 8930.281321] Key type .llcrypt unregistered [ 8930.285943] Key type ._llcrypt unregistered [ 8961.479146] Key type ._llcrypt registered [ 8961.480869] Key type .llcrypt registered [ 8961.611738] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 8962.907971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8962.989218] alg: No test for adler32 (adler32-zlib) [ 8964.024230] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 8964.311682] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 8966.047783] Key type lgssc registered [ 8967.353575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8976.039158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8976.065036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.682411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.717457] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 8987.107231] LustreError: 272810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8987.132873] LustreError: 272810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8992.228408] LustreError: 272809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8992.287626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8992.818763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8998.140396] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9006.659689] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9012.705465] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9021.512672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9022.085300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9023.184386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9029.349868] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9038.161109] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9068.536496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.542755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.544455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9068.546644] Lustre: Skipped 1 previous similar message [ 9068.575821] Lustre: Skipped 1 previous similar message [ 9070.957531] Lustre: server umount lustre-OST0000 complete [ 9078.754309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9078.764817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9078.772811] Lustre: Skipped 1 previous similar message [ 9082.289172] Lustre: server umount lustre-MDT0000 complete [ 9083.874684] LustreError: 272806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.904625] LustreError: 272806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9086.698911] LustreError: 272790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786698159 with bad export cookie 12841792260170194337 [ 9086.713345] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9086.723029] LustreError: 272790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9087.095382] Lustre: server umount lustre-MDT0001 complete [ 9100.554249] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9103.662684] Key type lgssc unregistered [ 9104.040567] LNet: 277131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9104.051355] LNetError: 277131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9104.094360] LNet: Removed LNI 192.168.202.106@tcp [ 9105.202204] Key type .llcrypt unregistered [ 9105.204255] Key type ._llcrypt unregistered [ 9125.132907] Key type ._llcrypt registered [ 9125.136686] Key type .llcrypt registered [ 9125.332134] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9126.825102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9126.850776] alg: No test for adler32 (adler32-zlib) [ 9127.967829] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 9128.170169] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9129.887192] Key type lgssc registered [ 9131.120855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9139.397227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9139.437812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9141.056778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.821612] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9151.471607] LustreError: 278370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9151.499202] LustreError: 278370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9156.597231] LustreError: 278372:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9156.894156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.365252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9163.379731] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9175.063151] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9183.907093] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9193.394153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9193.892915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9201.853553] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9211.338440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9211.524367] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9226.726685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9226.734590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9226.737087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9231.851424] LustreError: 280218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.944511] Lustre: server umount lustre-OST0000 complete [ 9236.963481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.970288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.978471] Lustre: Skipped 2 previous similar messages [ 9236.982043] Lustre: Skipped 1 previous similar message [ 9242.080983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9242.098389] Lustre: Skipped 2 previous similar messages [ 9243.317793] Lustre: server umount lustre-MDT0000 complete [ 9247.249804] LustreError: 278374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.281320] LustreError: 278374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9248.381041] LustreError: 278337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786698321 with bad export cookie 16989030281025593750 [ 9248.387919] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9248.397075] LustreError: 278337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9248.850714] Lustre: server umount lustre-MDT0001 complete [ 9261.239953] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9264.810060] Key type lgssc unregistered [ 9265.172816] LNet: 282076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9265.177912] LNetError: 282076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9265.203045] LNet: Removed LNI 192.168.202.106@tcp [ 9266.554872] Key type .llcrypt unregistered [ 9266.556990] Key type ._llcrypt unregistered [ 9280.044707] Key type ._llcrypt registered [ 9280.048855] Key type .llcrypt registered [ 9280.255855] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9298.457579] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9315.551028] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9316.829947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9316.974507] alg: No test for adler32 (adler32-zlib) [ 9318.070238] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 9318.322835] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9320.015802] Key type lgssc registered [ 9321.097839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9363.975639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9364.007786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.384971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9365.427611] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9365.533834] Lustre: lustre-MDT0000: new disk, initializing [ 9365.629697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.654287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9371.221301] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9384.302481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9384.458677] Lustre: 285631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9384.547142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9384.552386] Lustre: Skipped 1 previous similar message [ 9384.610423] Lustre: lustre-MDT0001: new disk, initializing [ 9384.726549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9384.781562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9384.810401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9390.788216] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9403.683152] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9410.789141] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9421.174940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9421.332788] Lustre: lustre-OST0000: new disk, initializing [ 9421.334690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9421.337812] Lustre: 287602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9421.428513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9429.763250] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9430.630228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9430.643899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9430.824154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9440.564898] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9446.560447] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9446.882389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9454.103544] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9454.350988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9456.611984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9456.616304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.619463] LustreError: Skipped 1 previous similar message [ 9456.621114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9456.650212] Lustre: Skipped 1 previous similar message [ 9461.738104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9461.743863] Lustre: Skipped 1 previous similar message [ 9462.908192] Lustre: server umount lustre-OST0000 complete [ 9471.969621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.992426] Lustre: Skipped 1 previous similar message [ 9471.995716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9472.015994] Lustre: Skipped 2 previous similar messages [ 9474.578849] Lustre: server umount lustre-MDT0000 complete [ 9479.225030] LustreError: 285625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786698552 with bad export cookie 2061861247623472950 [ 9479.229416] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9479.236189] LustreError: 285625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9479.533835] Lustre: server umount lustre-MDT0001 complete [ 9490.793637] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:09:22 (1786698562) [ 9498.082539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9498.550476] LustreError: 290014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.579592] LustreError: 290014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9498.692895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.632981] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9503.714463] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9508.843163] LustreError: 290014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9513.956544] LustreError: 290015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9514.343461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.707342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9521.671824] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9532.391182] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9539.459160] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9547.869428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9548.422182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9555.662676] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9565.990318] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9584.626356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9584.635386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.661480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9584.665851] Lustre: Skipped 1 previous similar message [ 9589.729061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9589.742731] Lustre: Skipped 1 previous similar message [ 9590.538118] Lustre: server umount lustre-OST0000 complete [ 9596.896508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9596.902134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.937548] Lustre: Skipped 1 previous similar message [ 9599.969167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9599.979212] Lustre: Skipped 2 previous similar messages [ 9601.437895] Lustre: server umount lustre-MDT0000 complete [ 9605.101041] LustreError: 290010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.140443] LustreError: 290010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9605.532159] LustreError: 289994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786698678 with bad export cookie 2061861247623473839 [ 9605.536064] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9605.545236] LustreError: 289994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9606.293875] Lustre: server umount lustre-MDT0001 complete [ 9618.694859] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [ 9622.189965] Key type lgssc unregistered [ 9622.602767] LNet: 293575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9622.611278] LNetError: 293575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9622.630922] LNet: Removed LNI 192.168.202.106@tcp [ 9623.889143] Key type .llcrypt unregistered [ 9623.892350] Key type ._llcrypt unregistered [ 9653.565335] Key type ._llcrypt registered [ 9653.567586] Key type .llcrypt registered [ 9653.646753] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9669.063602] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9670.096861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9670.105677] alg: No test for adler32 (adler32-zlib) [ 9671.180068] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [ 9671.437461] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [ 9673.111426] Key type lgssc registered [ 9674.154320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9725.533732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9725.568371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.912718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9726.943819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9727.002457] Lustre: lustre-MDT0000: new disk, initializing [ 9727.179145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.228730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9732.594723] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9744.317329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9744.502079] Lustre: 297404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9744.592095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9744.597459] Lustre: Skipped 1 previous similar message [ 9744.684620] Lustre: lustre-MDT0001: new disk, initializing [ 9744.816692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9744.907675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9744.928642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9751.292756] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9762.041695] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9768.957983] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9778.232646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9778.755128] Lustre: lustre-OST2710: new disk, initializing [ 9778.768906] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9778.777120] Lustre: 299373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9779.007143] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9787.146403] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9788.513097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9788.521276] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9788.679753] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9797.720106] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9804.743442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.968760] Lustre: lustre-OST03e8: new disk, initializing [ 9804.975661] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9804.979950] Lustre: 300439:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9805.089816] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9811.064839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9811.088931] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9811.217351] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9813.117870] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9821.217874] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9827.631148] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9827.782712] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9832.682717] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9832.909766] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9837.895242] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9838.098253] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9843.037052] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9843.468567] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9850.293862] Lustre: server umount lustre-MDT0000 complete [ 9852.390651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9852.398529] Lustre: Skipped 1 previous similar message [ 9852.408156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9853.727671] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786698926 with bad export cookie 18404443066360359852 [ 9853.739882] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9853.743841] LustreError: 297397:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9854.062561] Lustre: server umount lustre-MDT0001 complete [ 9869.502562] Lustre: server umount lustre-OST2710 complete [ 9873.375184] Lustre: 295204:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786698930/real 1786698930] req@00000000d8a9daf6 x1873489422259712/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786698946 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9873.406422] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9873.428451] Lustre: Skipped 2 previous similar messages [ 9873.951617] Lustre: server umount lustre-OST03e8 complete [ 9897.370256] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [ 9913.492904] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [ 9964.105581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9964.231964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9964.254347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9964.350263] Lustre: lustre-MDT0000: new disk, initializing [ 9964.473274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.497941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9969.835788] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9981.404315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.524879] Lustre: 305704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9981.572579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9981.575983] Lustre: Skipped 1 previous similar message [ 9981.645353] Lustre: lustre-MDT0001: new disk, initializing [ 9981.735719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.774952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9981.790860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9985.868578] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [ 9996.927915] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10003.947050] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10015.706463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10016.322969] Lustre: lustre-OST0000: new disk, initializing [10016.332292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10016.341685] Lustre: 307678:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10016.546871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10022.620286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10022.636918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10022.819596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10027.729928] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10039.894377] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10046.329385] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10046.982086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10053.070522] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10053.350263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10058.730679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10058.734852] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10058.758528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10058.764981] Lustre: Skipped 1 previous similar message [10058.791108] Lustre: Skipped 1 previous similar message [10062.152528] Lustre: server umount lustre-OST0000 complete [10068.961650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.964125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10068.987271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10068.994546] LustreError: Skipped 1 previous similar message [10069.008314] Lustre: Skipped 1 previous similar message [10073.590758] Lustre: server umount lustre-MDT0000 complete [10074.081585] LustreError: 306596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.089268] LustreError: 306596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [10078.097379] LustreError: 305696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699151 with bad export cookie 18404443066360361532 [10078.106410] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10078.126256] LustreError: 305696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10078.730121] Lustre: server umount lustre-MDT0001 complete [10088.094195] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:19:19 (1786699159) [10089.570908] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10091.234937] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:19:23 (1786699163) [10094.585994] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10122.151468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10122.480772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10122.501629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10122.659929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10128.007776] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10136.813802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10136.888708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10136.919200] Lustre: 311535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10142.299286] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10164.591478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.750577] Lustre: Denying initial registration attempt from nid 192.168.202.106@tcp, specified as failover [10164.759792] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10164.768570] LustreError: 312989:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10164.774375] LustreError: 312989:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10164.778663] LustreError: 312989:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10164.814923] Lustre: server umount lustre-OST0000 complete [10164.816990] LustreError: 312989:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10172.901706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10172.912780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.928609] Lustre: Skipped 1 previous similar message [10172.938095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10177.858448] Lustre: server umount lustre-MDT0000 complete [10178.531895] LustreError: 311547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.551293] LustreError: 311547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10181.789070] LustreError: 311530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699254 with bad export cookie 18404443066360362414 [10181.792928] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10181.800018] LustreError: 311530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10182.493696] Lustre: server umount lustre-MDT0001 complete [10194.865972] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10197.577157] Key type lgssc unregistered [10197.915442] LNet: 314116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10197.920345] LNetError: 314116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10197.930886] LNet: Removed LNI 192.168.202.106@tcp [10198.646925] Key type .llcrypt unregistered [10198.648849] Key type ._llcrypt unregistered [10214.034431] Key type ._llcrypt registered [10214.037301] Key type .llcrypt registered [10214.146315] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10215.604385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10215.674560] alg: No test for adler32 (adler32-zlib) [10216.856132] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [10217.166622] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10218.847635] Key type lgssc registered [10220.192672] Lustre: Echo OBD driver; http://www.lustre.org/ [10277.347741] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10289.767070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10289.800722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10291.259847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10291.306291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10291.597168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10297.337511] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10308.311892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10308.420376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10308.478443] Lustre: 316888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10309.040973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.502380] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10323.632559] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10331.472725] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10352.239576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.500709] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10352.685947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10359.237222] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10368.006793] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10371.553550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10371.565873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.596680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.077993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10372.089328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10372.111935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10372.119312] Lustre: Skipped 1 previous similar message [10376.814416] Lustre: server umount lustre-OST0000 complete [10382.304439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10382.321614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10382.324924] Lustre: Skipped 1 previous similar message [10387.500273] Lustre: server umount lustre-MDT0000 complete [10391.593855] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699464 with bad export cookie 12366071178628476588 [10391.601682] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10391.606610] LustreError: 316878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10392.181350] Lustre: server umount lustre-MDT0001 complete [10404.502542] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10407.452276] Key type lgssc unregistered [10407.805585] LNet: 320461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10407.821480] LNetError: 320461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10407.845625] LNet: Removed LNI 192.168.202.106@tcp [10408.805839] Key type .llcrypt unregistered [10408.811912] Key type ._llcrypt unregistered [10434.720530] Key type ._llcrypt registered [10434.722648] Key type .llcrypt registered [10434.839541] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10436.437688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10436.480523] alg: No test for adler32 (adler32-zlib) [10437.618903] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [10437.856925] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10439.495428] Key type lgssc registered [10440.510342] Lustre: Echo OBD driver; http://www.lustre.org/ [10448.897767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10448.940535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.464303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10454.483868] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10460.642390] LustreError: 321879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.663378] LustreError: 321879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10462.964581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.361565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10467.920371] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10475.564249] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.215610] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10492.588407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10493.110436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10500.024666] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10507.096703] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10519.521342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10519.526973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.536258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10520.044581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10520.048936] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10520.075776] Lustre: Skipped 1 previous similar message [10524.625363] Lustre: server umount lustre-MDT0000 complete [10525.163186] LustreError: 321879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10525.181870] LustreError: 321879:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10529.486803] LustreError: 321860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699602 with bad export cookie 12649524922586766972 [10529.487685] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10529.502682] LustreError: 321860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10530.272580] LustreError: 321873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.277739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.290056] LustreError: 321873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10530.327459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10530.330360] Lustre: Skipped 3 previous similar messages [10534.304206] LustreError: 322920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10534.321575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10534.342806] LustreError: 322920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10534.380669] Lustre: Skipped 1 previous similar message [10536.316901] Lustre: server umount lustre-MDT0001 complete [10544.208899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10556.169146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.678875] LustreError: 323710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.704503] LustreError: 323710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10574.794835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10574.859846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10579.702685] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10584.036627] LustreError: 325475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10584.048642] LustreError: 325475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10588.453835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.806534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10589.883677] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10594.321450] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10603.905091] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10606.319147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10613.671840] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10616.234092] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10624.482169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10624.491499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.509665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10624.513315] Lustre: Skipped 1 previous similar message [10626.030350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10627.482376] Lustre: server umount lustre-OST0000 complete [10634.720272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10634.737107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10634.760298] Lustre: Skipped 1 previous similar message [10634.780391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.793228] Lustre: Skipped 2 previous similar messages [10638.842466] Lustre: server umount lustre-MDT0000 complete [10641.408289] LustreError: 327348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.442567] LustreError: 327348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10643.277584] LustreError: 324664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699716 with bad export cookie 12649524922586774560 [10643.281950] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10643.290694] LustreError: 324664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10643.778441] Lustre: server umount lustre-MDT0001 complete [10658.634871] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10663.094671] Key type lgssc unregistered [10663.584800] LNet: 328156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10663.597972] LNetError: 328156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10663.659988] LNet: Removed LNI 192.168.202.106@tcp [10664.806935] Key type .llcrypt unregistered [10664.809206] Key type ._llcrypt unregistered [10694.632771] Key type ._llcrypt registered [10694.636649] Key type .llcrypt registered [10694.789509] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [10696.194512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10696.467216] alg: No test for adler32 (adler32-zlib) [10697.584825] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [10697.904845] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [10699.609586] Key type lgssc registered [10701.289962] Lustre: Echo OBD driver; http://www.lustre.org/ [10711.145844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10711.171426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.698483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10717.781912] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10722.275792] LustreError: 329579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.287463] LustreError: 329579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10726.657485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10727.183245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10731.764697] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10739.360947] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10745.810891] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10752.935901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.937414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10752.964529] Lustre: Skipped 1 previous similar message [10752.981799] Lustre: Skipped 1 previous similar message [10758.115685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10758.122742] Lustre: Skipped 1 previous similar message [10763.238677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10763.244045] Lustre: Skipped 1 previous similar message [10764.255814] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10764.410231] Lustre: server umount lustre-MDT0000 complete [10768.353519] LustreError: 329579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.375197] LustreError: 329579:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10768.502962] LustreError: 329559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699841 with bad export cookie 15041780262636170490 [10768.511835] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.844508] Lustre: server umount lustre-MDT0001 complete [10784.615356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10784.957703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10784.981268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10785.231378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10790.233282] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10800.234497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.468679] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10800.506877] Lustre: 332170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10800.778633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10806.206847] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10815.216300] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10821.619982] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10832.682068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10832.846027] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10832.854772] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10832.862178] LustreError: 334054:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10832.872572] LustreError: 334054:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10832.877986] LustreError: 334054:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10832.948821] Lustre: server umount lustre-OST0000 complete [10832.955385] LustreError: 334054:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10843.399098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10843.661908] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10843.818561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10851.121267] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10852.507320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10859.582525] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10868.859041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10868.906320] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10868.943802] Lustre: lustre-OST0001: new disk, initializing [10868.947418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10868.954453] Lustre: 335595:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10869.019292] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10875.508840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [10878.613735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10878.632205] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10878.753914] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10883.778695] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10889.185456] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10889.192770] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10889.201902] Lustre: Skipped 1 previous similar message [10889.204385] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10889.225145] Lustre: Skipped 1 previous similar message [10892.851337] Lustre: server umount lustre-OST0001 complete [10894.311027] LustreError: 334522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.340984] LustreError: 334522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10898.402394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10898.411291] LustreError: Skipped 1 previous similar message [10898.416253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10898.435422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10899.430691] LustreError: 334520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.450220] LustreError: 334520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10903.815083] Lustre: server umount lustre-OST0000 complete [10908.128415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10908.137855] LustreError: Skipped 1 previous similar message [10908.139898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10908.157477] Lustre: Skipped 1 previous similar message [10908.172163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10908.186179] Lustre: Skipped 3 previous similar messages [10913.809415] Lustre: server umount lustre-MDT0000 complete [10914.794401] LustreError: 332181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10914.812565] LustreError: 332181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10917.705304] LustreError: 332161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786699990 with bad export cookie 15041780262636171043 [10917.706635] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10917.720077] LustreError: 332161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10918.115255] Lustre: server umount lustre-MDT0001 complete [10929.434508] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [10932.523463] Key type lgssc unregistered [10932.863629] LNet: 337558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10932.870413] LNetError: 337558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10932.882280] LNet: Removed LNI 192.168.202.106@tcp [10933.703301] Key type .llcrypt unregistered [10933.707379] Key type ._llcrypt unregistered [11006.076856] Key type ._llcrypt registered [11006.079107] Key type .llcrypt registered [11006.199425] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [11021.287631] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11022.629935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11022.662165] alg: No test for adler32 (adler32-zlib) [11023.809254] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [11024.112333] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11025.871178] Key type lgssc registered [11026.768787] Lustre: Echo OBD driver; http://www.lustre.org/ [11061.912647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11061.948055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11063.239604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11063.260570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11063.331130] Lustre: lustre-MDT0000: new disk, initializing [11063.396453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11063.409688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11068.529489] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11082.192343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11082.435294] Lustre: 342496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11082.498282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11082.502268] Lustre: Skipped 1 previous similar message [11082.558882] Lustre: lustre-MDT0001: new disk, initializing [11082.675637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11082.724398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11082.744052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11088.073409] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11098.732065] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11105.138098] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11112.751233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11113.098308] Lustre: lustre-OST0000: new disk, initializing [11113.105395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11113.117930] Lustre: 344467:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11113.246479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11119.705553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11119.717587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11119.817065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11120.304985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11129.369319] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11133.826922] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11134.008793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11138.632243] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11138.887866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11145.701223] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.718782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.726029] Lustre: Skipped 1 previous similar message [11145.728328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11145.759560] Lustre: Skipped 1 previous similar message [11147.245468] Lustre: server umount lustre-OST0000 complete [11154.401311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11154.407187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.428343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.940565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11155.942732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11157.852083] Lustre: server umount lustre-MDT0000 complete [11162.120316] LustreError: 342490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700235 with bad export cookie 2078465278574139335 [11162.134484] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11162.142550] LustreError: 342490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11162.533240] Lustre: server umount lustre-MDT0001 complete [11172.977886] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:37:24 (1786700244) [11181.167446] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:37:33 (1786700253) [11190.920430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11191.252822] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11191.269555] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11191.334890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11196.012919] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11196.386193] LustreError: 347223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11201.508431] LustreError: 347222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11205.597191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11205.902683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11209.625972] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11216.257347] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11222.382844] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11229.483114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11229.889070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11236.079319] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11243.744890] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.013510] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11254.276838] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11256.357934] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11258.636561] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11261.071660] Lustre: Failing over lustre-MDT0000 [11261.337300] Lustre: server umount lustre-MDT0000 complete [11261.930910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11261.970930] Lustre: Skipped 1 previous similar message [11261.977213] LustreError: 349065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11262.018520] LustreError: 349065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11267.043195] LustreError: 348315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.088288] LustreError: 348315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11277.292573] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11277.314474] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [11281.707854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11281.898651] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11282.215788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.313284] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11283.523274] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11287.528675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11287.619951] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11287.696050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11287.917339] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11298.721777] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11300.574571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11303.921809] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11305.544170] Lustre: Failing over lustre-MDT0000 [11307.773891] Lustre: server umount lustre-MDT0000 complete [11308.009619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11308.010938] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.023837] Lustre: Skipped 2 previous similar messages [11308.038191] LustreError: 347217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11308.043683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11324.385504] Lustre: 340721:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786700381/real 1786700381] req@00000000bbadacbd x1873490841003264/t0(0) o400->MGC192.168.202.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786700397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11324.418965] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.757265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11335.095505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11335.201314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11339.839996] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11340.271860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11340.279295] Lustre: Skipped 2 previous similar messages [11340.312756] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11340.358942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11340.455332] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11351.674375] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11353.587187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11360.741552] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11360.751792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11360.760557] Lustre: Skipped 1 previous similar message [11360.764313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11360.783706] Lustre: Skipped 3 previous similar messages [11365.597964] Lustre: server umount lustre-MDT0000 complete [11365.866218] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11365.899216] LustreError: 347218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [11369.184292] LustreError: 347204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700442 with bad export cookie 2078465278574142044 [11369.185769] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11369.193345] LustreError: 347204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11369.475956] Lustre: server umount lustre-MDT0001 complete [11383.650276] Lustre: server umount lustre-OST0000 complete [11399.812866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11400.138854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11404.082627] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11413.396420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11419.487368] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11428.858738] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11435.084101] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11442.600661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11445.112303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11450.007438] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11457.761732] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11463.648457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11463.650895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11463.664759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11463.672027] Lustre: Skipped 3 previous similar messages [11463.682600] Lustre: Skipped 1 previous similar message [11465.397631] Lustre: server umount lustre-OST0000 complete [11473.888098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11473.895430] Lustre: Skipped 2 previous similar messages [11475.859499] Lustre: server umount lustre-MDT0000 complete [11479.012326] LustreError: 354292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11479.039220] LustreError: 354292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11480.174274] LustreError: 353545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700553 with bad export cookie 2078465278574142793 [11480.186403] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11480.653972] Lustre: server umount lustre-MDT0001 complete [11493.133600] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11495.600579] Key type lgssc unregistered [11495.837604] LNet: 357098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11495.843046] LNetError: 357098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11496.875534] LNet: Removed LNI 192.168.202.106@tcp [11497.674904] Key type .llcrypt unregistered [11497.677380] Key type ._llcrypt unregistered [11523.956813] Key type ._llcrypt registered [11523.958459] Key type .llcrypt registered [11524.047813] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11525.335712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11525.360945] alg: No test for adler32 (adler32-zlib) [11526.456904] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [11526.710540] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11528.471633] Key type lgssc registered [11529.545209] Lustre: Echo OBD driver; http://www.lustre.org/ [11537.183997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11537.200172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.915996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11542.461167] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11548.133080] LustreError: 358617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11548.144991] LustreError: 358617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11551.623955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.878221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11556.964981] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11564.637918] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11570.094190] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11576.635074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11576.867276] LustreError: 360435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.062494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11583.473408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11583.649357] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11591.191410] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11608.040760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11608.050198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11608.081178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11609.060441] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11609.078356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11609.083820] Lustre: Skipped 1 previous similar message [11609.108447] Lustre: Skipped 2 previous similar messages [11614.176745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11619.297619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11619.300166] Lustre: Skipped 2 previous similar messages [11620.319669] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11620.612562] Lustre: server umount lustre-MDT0000 complete [11624.420869] LustreError: 358613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11624.464415] LustreError: 358613:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11625.013193] LustreError: 358598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700698 with bad export cookie 14462321426272727553 [11625.033752] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11625.356590] Lustre: server umount lustre-MDT0001 complete [11640.137663] Lustre: server umount lustre-OST0000 complete [11652.979847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.560405] LustreError: 362350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11653.639103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11658.738605] LustreError: 362351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11659.393822] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11668.968715] LustreError: 362350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11668.998126] LustreError: 362350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11669.097067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.504846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11673.989561] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11682.550047] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11690.527382] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11699.567030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11699.921195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11707.405878] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11715.738383] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11723.509242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11739.105568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.110340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11739.121147] Lustre: Skipped 2 previous similar messages [11739.135436] Lustre: Skipped 2 previous similar messages [11745.234980] Lustre: server umount lustre-MDT0000 complete [11749.359566] LustreError: 362351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.372053] LustreError: 362330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700822 with bad export cookie 14462321426272728666 [11749.385238] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11749.390644] LustreError: 362351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11749.446861] LustreError: 362330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11749.959480] Lustre: server umount lustre-MDT0001 complete [11763.285286] Lustre: server umount lustre-OST0000 complete [11778.986049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.542199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11784.121049] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11784.673531] LustreError: 366284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11784.707565] LustreError: 366284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11793.276359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11797.895898] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11805.985294] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11811.627424] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11817.358587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11817.591692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11817.597594] Lustre: Skipped 1 previous similar message [11822.916915] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [11824.748096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11829.990726] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11835.362131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11835.368054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11835.372934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11835.372943] Lustre: Skipped 3 previous similar messages [11835.377600] Lustre: Skipped 1 previous similar message [11841.807713] Lustre: server umount lustre-OST0000 complete [11850.212087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11850.229862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.857296] Lustre: server umount lustre-MDT0000 complete [11855.845993] LustreError: 366285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11855.885181] LustreError: 366285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11857.577460] LustreError: 366266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786700930 with bad export cookie 14462321426272729688 [11857.586653] LustreError: 366266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11857.586841] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11858.032384] Lustre: server umount lustre-MDT0001 complete [11872.849877] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11876.730509] Key type lgssc unregistered [11877.187643] LNet: 369821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11877.200276] LNetError: 369821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11877.219953] LNet: Removed LNI 192.168.202.106@tcp [11878.495174] Key type .llcrypt unregistered [11878.497518] Key type ._llcrypt unregistered [11906.863437] Key type ._llcrypt registered [11906.865648] Key type .llcrypt registered [11906.977670] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11907.723971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11907.754512] alg: No test for adler32 (adler32-zlib) [11908.852529] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [11909.173906] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11910.912721] Key type lgssc registered [11911.951737] Lustre: Echo OBD driver; http://www.lustre.org/ [11920.730642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11920.758222] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11920.768566] LustreError: 371262:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11920.788423] LustreError: 371262:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11920.794600] LustreError: 371262:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11920.802674] LustreError: 371262:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11920.811591] LustreError: 371262:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11929.963699] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11929.975955] LustreError: 371659:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11930.024443] LustreError: 371659:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11930.039736] LustreError: 371659:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11930.044229] LustreError: 371659:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11930.060354] LustreError: 371659:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11952.330591] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [11955.588636] Key type lgssc unregistered [11955.848394] LNet: 372666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11955.852880] LNetError: 372666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11955.868723] LNet: Removed LNI 192.168.202.106@tcp [11956.575543] Key type .llcrypt unregistered [11956.577883] Key type ._llcrypt unregistered [11977.054934] Key type ._llcrypt registered [11977.058633] Key type .llcrypt registered [11977.187892] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [11993.733943] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [11994.799963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11994.880349] alg: No test for adler32 (adler32-zlib) [11996.050820] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [11996.304255] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [11997.959215] Key type lgssc registered [11999.026729] Lustre: Echo OBD driver; http://www.lustre.org/ [12038.223713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12038.263257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12039.542343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12039.568469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12039.653745] Lustre: lustre-MDT0000: new disk, initializing [12039.797444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12039.825380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12044.594661] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12055.797694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12055.917662] Lustre: 375832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12056.003567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12056.013203] Lustre: Skipped 1 previous similar message [12056.097722] Lustre: lustre-MDT0001: new disk, initializing [12056.231871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12056.293828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12056.312164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12060.812537] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12071.277876] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12078.185092] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12085.123589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12085.477242] Lustre: lustre-OST0000: new disk, initializing [12085.485962] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12085.496136] Lustre: 377797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12085.613191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12092.130485] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12093.517763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12093.538224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12093.681044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12102.763622] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12107.351525] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12107.612850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12112.425668] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12112.627965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12119.520980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12119.538847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12119.547854] LustreError: Skipped 1 previous similar message [12119.550781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12119.592654] Lustre: Skipped 1 previous similar message [12120.892300] Lustre: server umount lustre-OST0000 complete [12127.715839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12127.728655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.745169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12127.753746] Lustre: Skipped 1 previous similar message [12129.762373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12129.763629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.258330] Lustre: server umount lustre-MDT0000 complete [12135.219068] LustreError: 375823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786701208 with bad export cookie 1019934011544785689 [12135.228475] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12135.229317] LustreError: 375823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12135.647472] Lustre: server umount lustre-MDT0001 complete [12145.431357] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12147.606753] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:53:39 (1786701219) [12155.125517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12155.486073] LustreError: 380258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12155.497762] LustreError: 380258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12155.546217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12160.416532] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12160.999873] LustreError: 380259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.093823] LustreError: 380258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12170.211993] LustreError: 380259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.242325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.759475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12178.772903] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12189.685693] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12197.980802] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12206.351548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12206.735401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12212.758978] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12220.389223] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12228.384866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12228.600742] Lustre: lustre-OST0001: new disk, initializing [12228.622908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12228.635299] Lustre: 383095:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12228.723722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12235.701672] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12237.427482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12237.440716] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12237.590459] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12246.411252] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12253.154767] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12253.160488] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12253.160932] LustreError: Skipped 1 previous similar message [12253.170602] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12253.170610] Lustre: Skipped 1 previous similar message [12253.219318] Lustre: Skipped 1 previous similar message [12257.093670] Lustre: server umount lustre-OST0001 complete [12258.287139] LustreError: 382912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.332614] LustreError: 382912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12268.513842] LustreError: 382912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.569065] LustreError: 382912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12276.710623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12276.718761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12276.730265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12276.738741] Lustre: Skipped 2 previous similar messages [12282.749357] Lustre: server umount lustre-OST0000 complete [12288.994069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12289.002188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12289.013623] Lustre: Skipped 2 previous similar messages [12289.041161] Lustre: Skipped 3 previous similar messages [12301.279874] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12301.740550] Lustre: server umount lustre-MDT0000 complete [12303.328896] LustreError: 380258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12303.350707] LustreError: 380258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12305.870238] LustreError: 380239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786701378 with bad export cookie 1019934011544786571 [12305.879591] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.893447] LustreError: 380239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12312.329241] Lustre: server umount lustre-MDT0001 complete [12325.250439] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12328.009346] Key type lgssc unregistered [12328.367702] LNet: 385085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12328.374268] LNetError: 385085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12328.402526] LNet: Removed LNI 192.168.202.106@tcp [12329.493183] Key type .llcrypt unregistered [12329.499973] Key type ._llcrypt unregistered [12385.155805] Key type ._llcrypt registered [12385.159595] Key type .llcrypt registered [12385.281268] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12386.962577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12387.009632] alg: No test for adler32 (adler32-zlib) [12388.059991] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [12388.336374] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12390.055188] Key type lgssc registered [12391.249049] Lustre: Echo OBD driver; http://www.lustre.org/ [12399.539945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12399.566623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12400.846675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12400.868737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12401.034427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12405.909531] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12415.265427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12415.348505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12415.375525] Lustre: 387886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12415.598138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12420.713274] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12429.381649] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12437.131979] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12441.056261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12441.064337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12441.078476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12441.570106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12441.599204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12446.690234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12446.696605] Lustre: Skipped 1 previous similar message [12451.810743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12451.820292] Lustre: Skipped 1 previous similar message [12454.367112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12454.680020] Lustre: server umount lustre-MDT0000 complete [12458.743512] LustreError: 387879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786701531 with bad export cookie 17371628192012614711 [12458.759664] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12458.765469] LustreError: 387879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12459.211604] Lustre: server umount lustre-MDT0001 complete [12471.432936] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:59:02 (1786701542) [12483.324391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.179815] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12484.220142] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12484.300756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12489.699579] LustreError: 390400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12491.339662] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12494.843122] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12499.946215] LustreError: 390400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.351249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12502.590543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12508.073795] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12517.820809] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12524.672713] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12531.914846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12532.252719] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12532.534737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12540.325828] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12552.909452] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12563.621315] Lustre: Permanently deactivating lustre-OST0000 [12563.677854] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12573.238845] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12576.203762] LustreError: 393438:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12576.213443] LustreError: 393438:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12576.219634] LustreError: 393438:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12579.328686] LustreError: 393488:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12579.344559] LustreError: 393488:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12587.932956] Lustre: server umount lustre-OST0000 complete [12591.367925] LustreError: 393749:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12591.376387] LustreError: 393749:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12594.657114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12594.665100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12594.677192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12596.705274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12604.083156] Lustre: server umount lustre-MDT0000 complete [12606.945065] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12606.970034] LustreError: 390399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12609.690489] LustreError: 390379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786701682 with bad export cookie 17371628192012615292 [12609.701268] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12609.718963] LustreError: 390379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12619.470596] Lustre: server umount lustre-MDT0001 complete [12629.476191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.298451] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12638.300915] LustreError: 394909:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12644.739946] Lustre: 395249:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.106@tcp [12646.921692] Lustre: server umount MGS complete [12659.642552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12660.291175] LustreError: 395973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12660.323317] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12660.334345] Lustre: Skipped 1 previous similar message [12660.382167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12664.644690] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12674.306557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12674.838613] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12681.697145] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12691.013744] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12697.249333] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12699.099565] Lustre: Permanently reactivating lustre-OST0000 [12699.118571] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12699.124900] Lustre: Skipped 2 previous similar messages [12728.287305] LustreError: 397802:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12736.724941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12737.004565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12737.009881] Lustre: Skipped 1 previous similar message [12743.491945] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12751.724861] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12757.473094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12757.483045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12757.496737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12757.503713] Lustre: Skipped 4 previous similar messages [12759.054594] Lustre: server umount lustre-OST0000 complete [12768.562879] Lustre: server umount lustre-MDT0000 complete [12768.736390] LustreError: 395974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.754616] LustreError: 395974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12773.021469] LustreError: 395953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786701846 with bad export cookie 17371628192012616342 [12773.040898] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12773.574726] Lustre: server umount lustre-MDT0001 complete [12785.819809] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [12788.475265] Key type lgssc unregistered [12788.793558] LNet: 400385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12788.812802] LNetError: 400385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12788.840989] LNet: Removed LNI 192.168.202.106@tcp [12789.903497] Key type .llcrypt unregistered [12789.905768] Key type ._llcrypt unregistered [12810.558275] Key type ._llcrypt registered [12810.561970] Key type .llcrypt registered [12810.742523] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [12828.537849] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [12830.214873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12830.248189] alg: No test for adler32 (adler32-zlib) [12831.359377] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [12831.623547] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [12833.311213] Key type lgssc registered [12834.657148] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.731810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.776250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.097900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12873.122147] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12873.217309] Lustre: lustre-MDT0000: new disk, initializing [12873.275835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12873.293883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12876.987676] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12887.218682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.361696] Lustre: 403544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12887.444462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12887.452112] Lustre: Skipped 1 previous similar message [12887.566254] Lustre: lustre-MDT0001: new disk, initializing [12887.673827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12887.760507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12887.776241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12892.184448] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12901.397228] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12907.134346] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12915.675326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12916.364436] Lustre: lustre-OST0000: new disk, initializing [12916.375349] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12916.387327] Lustre: 405516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12916.603932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12921.978109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12921.998854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12922.143067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12924.694914] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12935.979462] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12941.184743] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12941.372520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12945.706148] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12945.896835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12953.057498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12953.058720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12953.099728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12954.473069] Lustre: server umount lustre-OST0000 complete [12959.200877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12959.207777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12959.224124] Lustre: Skipped 1 previous similar message [12959.228628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12959.234400] Lustre: Skipped 1 previous similar message [12963.297333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.297377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12963.330973] Lustre: Skipped 1 previous similar message [12965.323784] Lustre: server umount lustre-MDT0000 complete [12969.862241] LustreError: 403538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786702042 with bad export cookie 1707290349531019454 [12969.869410] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12969.883166] LustreError: 403538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12970.696982] Lustre: server umount lustre-MDT0001 complete [12980.951371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12982.203342] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 06:07:34 (1786702054) [12989.793137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.371909] LustreError: 407979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12990.384782] LustreError: 407979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12990.418620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12994.665650] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [12995.559365] LustreError: 407980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.673695] LustreError: 407979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.418762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.651986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13008.302923] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13016.795632] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13022.644930] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13029.823047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13030.192579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13036.319366] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13045.035842] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13047.353907] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13051.705273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13059.040944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13059.042324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13059.054545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13059.066300] Lustre: Skipped 1 previous similar message [13059.081187] Lustre: Skipped 1 previous similar message [13064.160708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13064.555913] Lustre: server umount lustre-OST0000 complete [13069.280629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13069.301426] Lustre: Skipped 1 previous similar message [13074.401394] LustreError: 407974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13074.427450] LustreError: 407974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13074.466199] Lustre: server umount lustre-MDT0000 complete [13077.943367] LustreError: 407961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786702151 with bad export cookie 1707290349531020336 [13077.950933] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13077.979634] LustreError: 407961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13078.559413] Lustre: server umount lustre-MDT0001 complete [13090.386840] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13093.544366] Key type lgssc unregistered [13093.870246] LNet: 411708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13093.878962] LNetError: 411708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13094.887304] LNet: Removed LNI 192.168.202.106@tcp [13095.701680] Key type .llcrypt unregistered [13095.703946] Key type ._llcrypt unregistered [13124.773440] Key type ._llcrypt registered [13124.777271] Key type .llcrypt registered [13124.886158] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13125.358311] Key type .llcrypt unregistered [13125.370713] Key type ._llcrypt unregistered [13147.290918] Key type ._llcrypt registered [13147.293949] Key type .llcrypt registered [13147.443792] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13149.546825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13149.617190] alg: No test for adler32 (adler32-zlib) [13150.645963] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [13150.894276] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13152.551863] Key type lgssc registered [13153.948508] Lustre: Echo OBD driver; http://www.lustre.org/ [13162.890535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13162.908129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13164.395421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13169.051670] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13169.632141] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.659483] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13174.768492] LustreError: 413929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.486561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13178.954746] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.986807] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13179.007423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13183.970865] LustreError: 413929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.999427] LustreError: 413929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13185.640224] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13189.101213] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13189.123171] LustreError: 413928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13189.142216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13193.982208] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13199.330923] LustreError: 413925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.357917] LustreError: 413925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13202.041865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13202.501246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13207.309725] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13217.789539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13217.805726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13217.831743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13222.900565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13222.915986] Lustre: Skipped 2 previous similar messages [13223.259200] Lustre: server umount lustre-OST0000 complete [13228.001213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13228.015023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13228.019316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13228.031981] Lustre: Skipped 2 previous similar messages [13228.051089] Lustre: Skipped 1 previous similar message [13233.126449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13233.139323] Lustre: Skipped 1 previous similar message [13234.334098] Lustre: server umount lustre-MDT0000 complete [13238.240818] LustreError: 415666:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.267380] LustreError: 415666:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13238.843220] LustreError: 413908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786702311 with bad export cookie 13865733134690794703 [13238.853832] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13239.219474] Lustre: server umount lustre-MDT0001 complete [13254.030262] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13256.673433] Key type lgssc unregistered [13257.051953] LNet: 417079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13257.065795] LNetError: 417079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13258.095138] LNet: Removed LNI 192.168.202.106@tcp [13259.162630] Key type .llcrypt unregistered [13259.165642] Key type ._llcrypt unregistered [13290.031254] Key type ._llcrypt registered [13290.039064] Key type .llcrypt registered [13290.205276] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13292.601022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13292.669545] alg: No test for adler32 (adler32-zlib) [13294.091260] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [13294.835621] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13296.575190] Key type lgssc registered [13298.226596] Lustre: Echo OBD driver; http://www.lustre.org/ [13307.918198] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13307.951107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13323.488552] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13328.607434] LustreError: 418473:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.106@tcp: failed processing log, type 4: rc = -110 [13355.360401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13361.764833] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13371.103422] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13379.376344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.910783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13379.990053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13385.188555] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13385.223609] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13386.724148] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13390.316942] LustreError: 418500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13390.342485] LustreError: 418500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13395.424692] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13395.451178] LustreError: 418499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13399.004531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13399.707061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13404.035947] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13413.228129] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13421.238863] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13430.245692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13430.257753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13430.272219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13430.769645] LustreError: 418491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13430.906570] Lustre: server umount lustre-OST0000 complete [13435.074865] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13435.081201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13435.099788] Lustre: Skipped 1 previous similar message [13435.119936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13440.993880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13440.998332] Lustre: Skipped 2 previous similar messages [13441.451570] Lustre: server umount lustre-MDT0000 complete [13445.933130] LustreError: 418482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786702518 with bad export cookie 11709078489721537283 [13445.934089] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13445.949776] LustreError: 418482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13446.216840] Lustre: server umount lustre-MDT0001 complete [13458.883763] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13461.698203] Key type lgssc unregistered [13462.137788] LNet: 422047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13462.150638] LNetError: 422047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13463.209394] LNet: Removed LNI 192.168.202.106@tcp [13464.054630] Key type .llcrypt unregistered [13464.056213] Key type ._llcrypt unregistered [13493.256225] Key type ._llcrypt registered [13493.258575] Key type .llcrypt registered [13493.350669] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13494.171638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13494.199135] alg: No test for adler32 (adler32-zlib) [13495.532661] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [13495.845543] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13497.496541] Key type lgssc registered [13498.646948] Lustre: Echo OBD driver; http://www.lustre.org/ [13507.462496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13507.498542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13509.209146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13514.540048] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13519.337750] LustreError: 423465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13519.360026] LustreError: 423465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13524.459639] LustreError: 423466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13524.551822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13524.905663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13529.949604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13538.286731] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13544.164882] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13552.329958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13552.738473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13554.833192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13558.911809] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13568.423568] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13574.640609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13574.668876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13574.689330] Lustre: Skipped 2 previous similar messages [13574.726546] Lustre: Skipped 2 previous similar messages [13575.795201] LustreError: 425411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.871508] Lustre: server umount lustre-MDT0000 complete [13579.078488] LustreError: 423446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.6@tcp arrived at 1786702652 with bad export cookie 17039792111211445067 [13579.097819] LustreError: 423446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13579.745078] LustreError: 423466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13579.780779] LustreError: 423466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13582.185356] LustreError: 426161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786702655 with bad export cookie 17039792111211444178 [13582.199976] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13584.885666] LustreError: 423461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.894970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13584.895259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13584.895302] LustreError: 423461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13584.944401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13587.873886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13588.587370] Lustre: server umount lustre-OST0000 complete [13596.618666] Lustre: server umount lustre-MDT0001 complete [13609.136552] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13612.050499] Key type lgssc unregistered [13612.300556] LNet: 427156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13612.307539] LNetError: 427156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13613.353936] LNet: Removed LNI 192.168.202.106@tcp [13614.223937] Key type .llcrypt unregistered [13614.226279] Key type ._llcrypt unregistered [13642.423690] Key type ._llcrypt registered [13642.427198] Key type .llcrypt registered [13642.507787] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13643.512045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13643.646277] alg: No test for adler32 (adler32-zlib) [13644.736993] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [13645.020368] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13646.687237] Key type lgssc registered [13647.883810] Lustre: Echo OBD driver; http://www.lustre.org/ [13655.830407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13655.856654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13657.495304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13663.203964] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13667.809085] LustreError: 428574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13667.837548] LustreError: 428574:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13672.717194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13672.936551] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13673.059100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13678.430265] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13686.479427] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13691.817337] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13699.480509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13700.007034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13702.075810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13707.206189] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13717.521913] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13723.622348] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13723.625141] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13723.645476] Lustre: Skipped 1 previous similar message [13723.658951] Lustre: Skipped 1 previous similar message [13725.288354] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [13728.755626] LustreError: 428569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.764842] LustreError: 428569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13728.857629] Lustre: server umount lustre-MDT0001 complete [13730.371662] LustreError: 428571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13733.859939] LustreError: 428570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13733.893991] LustreError: 428570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13737.955890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13737.963694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13737.979729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13738.981332] LustreError: 428574:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13742.439596] Lustre: server umount lustre-OST0000 complete [13747.118388] Lustre: server umount lustre-MDT0000 complete [13761.686836] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [13764.972345] Key type lgssc unregistered [13765.263192] LNet: 432260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13765.271024] LNetError: 432260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13765.291619] LNet: Removed LNI 192.168.202.106@tcp [13766.324810] Key type .llcrypt unregistered [13766.326776] Key type ._llcrypt unregistered [13801.916676] Key type ._llcrypt registered [13801.918759] Key type .llcrypt registered [13802.084521] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [13821.327289] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13822.642107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13822.694086] alg: No test for adler32 (adler32-zlib) [13823.801989] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [13824.066602] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [13825.751594] Key type lgssc registered [13826.764783] Lustre: Echo OBD driver; http://www.lustre.org/ [13874.376624] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [13893.146934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13893.186834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13894.449875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13894.468243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13894.534164] Lustre: lustre-MDT0000: new disk, initializing [13894.697907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13894.716563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13900.509532] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13919.238103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13919.489796] Lustre: lustre-OST0000: new disk, initializing [13919.496798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13919.506179] Lustre: Skipped 1 previous similar message [13919.518496] Lustre: 437340:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13919.696611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13926.060222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13926.074132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13926.123068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13927.170869] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13939.001417] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13961.663517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13962.003967] Lustre: 436321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13962.200836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13962.342261] Lustre: lustre-MDT0001: new disk, initializing [13962.551405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13962.632402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13962.660558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13967.423959] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [13977.157823] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13977.425848] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13981.977921] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13982.247470] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13990.491231] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.6@tcp (stopping) [13991.906273] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13991.910563] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13991.921122] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13991.929187] Lustre: Skipped 1 previous similar message [13991.945476] Lustre: Skipped 1 previous similar message [13993.558222] Lustre: server umount lustre-MDT0001 complete [14002.148957] LustreError: 436327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14002.184076] LustreError: 436327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14007.268181] LustreError: 436333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14007.288155] LustreError: 436333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14012.384373] LustreError: 436332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14012.401235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14012.410533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14017.509426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14017.509540] LustreError: 436333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.625090] LustreError: 436332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14022.625573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14025.183146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14025.344758] Lustre: server umount lustre-OST0000 complete [14044.127264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14044.518802] Lustre: server umount lustre-MDT0000 complete [14060.387490] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [14063.654642] Key type lgssc unregistered [14063.931600] LNet: 441095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14063.938919] LNetError: 441095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14063.969893] LNet: Removed LNI 192.168.202.106@tcp [14064.966115] Key type .llcrypt unregistered [14064.968653] Key type ._llcrypt unregistered [14112.462885] Key type ._llcrypt registered [14112.465951] Key type .llcrypt registered [14112.651834] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [14113.298288] Key type .llcrypt unregistered [14113.307093] Key type ._llcrypt unregistered [14134.664699] Key type ._llcrypt registered [14134.669783] Key type .llcrypt registered [14134.896379] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [14137.030356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14137.083134] alg: No test for adler32 (adler32-zlib) [14138.148583] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [14138.458593] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [14140.159168] Key type lgssc registered [14141.715725] Lustre: Echo OBD driver; http://www.lustre.org/ [14170.284548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14170.320364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14171.662889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14171.728508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14171.905068] Lustre: lustre-MDT0000: new disk, initializing [14171.986449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14172.006706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14176.153970] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14189.500226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14189.777533] Lustre: 444476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14190.043580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14190.047026] Lustre: Skipped 1 previous similar message [14190.163383] Lustre: lustre-MDT0001: new disk, initializing [14190.410115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14190.490565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14190.504349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14195.455482] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14207.760920] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14213.395515] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14220.035028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14220.241410] Lustre: lustre-OST0000: new disk, initializing [14220.244740] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14220.249324] Lustre: 446448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14220.328495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14226.634668] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14228.581121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14228.592947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14228.700570] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14237.274870] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14249.456072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14249.467856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14249.509138] Lustre: Skipped 2 previous similar messages [14249.522490] Lustre: Skipped 2 previous similar messages [14254.141453] Lustre: server umount lustre-MDT0000 complete [14259.683924] LustreError: 445582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14259.704763] LustreError: 445582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [14259.926482] LustreError: 444470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786703332 with bad export cookie 6871870905009261956 [14259.928123] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14259.933836] LustreError: 444470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14260.283457] Lustre: server umount lustre-MDT0001 complete [14275.905698] Lustre: server umount lustre-OST0000 complete [14287.232334] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:29:18 (1786703358) [14298.492031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14299.075552] LustreError: 448478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14299.287394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14304.239746] LustreError: 448479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14304.302431] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14309.351063] LustreError: 448478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14314.475042] LustreError: 448479:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14315.491909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14315.993485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14321.233307] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14330.544975] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14337.684100] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14360.367472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14360.609489] Lustre: Found index 0 for lustre-OST0000, updating log [14360.893872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14367.923291] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [14370.109425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14375.839903] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14385.317710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14385.335619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14386.408277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14389.518603] Lustre: server umount lustre-OST0000 complete [14408.162692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14408.183153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14408.185048] Lustre: Skipped 1 previous similar message [14408.203995] Lustre: Skipped 2 previous similar messages [14411.769914] Lustre: server umount lustre-MDT0000 complete [14412.324859] LustreError: 448475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14412.338733] LustreError: 448475:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14415.800682] LustreError: 448458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786703488 with bad export cookie 6871870905009263349 [14415.805621] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14415.810957] LustreError: 448458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14416.188597] Lustre: server umount lustre-MDT0001 complete [14425.510397] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:31:36 (1786703496) [14427.297733] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [15053.795858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15054.181764] LustreError: 454892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15054.201579] LustreError: 454892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15054.276378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15059.433544] LustreError: 454893:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15059.766441] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15064.545559] LustreError: 454892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15071.252229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15071.727631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15077.406328] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15087.263727] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15095.588329] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15105.183719] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [15116.601518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15116.914658] Lustre: Found index 0 for lustre-OST0000, updating log [15117.245292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15124.558496] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15133.167891] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15135.714988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15135.727929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15135.770560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15141.803318] Lustre: server umount lustre-OST0000 complete [15147.109347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15147.112795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15147.129769] Lustre: Skipped 1 previous similar message [15147.143501] Lustre: Skipped 1 previous similar message [15148.200728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.241785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15150.255585] Lustre: Skipped 3 previous similar messages [15152.492026] Lustre: server umount lustre-MDT0000 complete [15153.313028] LustreError: 455832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15153.329313] LustreError: 455832:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15156.205941] LustreError: 454874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786704229 with bad export cookie 6871870905009264413 [15156.207394] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15156.215061] LustreError: 454874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15156.540054] Lustre: server umount lustre-MDT0001 complete [15166.623500] LNet: 1 local NIs in recovery (showing 1): 192.168.202.106@tcp [15188.149267] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [15199.813957] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15242.928539] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:45:14 (1786704314) [15260.548632] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15263.221423] Key type lgssc unregistered [15263.499494] LNet: 461507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15263.503803] LNetError: 461507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15378.223609] LNet: Removed LNI 192.168.202.106@tcp [15379.118989] Key type .llcrypt unregistered [15379.123793] Key type ._llcrypt unregistered [15399.302303] Key type ._llcrypt registered [15399.304217] Key type .llcrypt registered [15399.414629] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15400.443970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15400.507070] alg: No test for adler32 (adler32-zlib) [15401.626284] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [15401.899425] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15403.543215] Key type lgssc registered [15404.809359] Lustre: Echo OBD driver; http://www.lustre.org/ [15409.800356] LNet: Added LNI 192.168.202.106@tcp100 [8/256/0/180] [15410.829985] LNet: Added LNI 192.168.202.106@tcp101 [8/256/0/180] [15411.841700] LNet: Added LNI 192.168.202.106@tcp102 [8/256/0/180] [15412.950660] LNet: Added LNI 192.168.202.106@tcp103 [8/256/0/180] [15413.785082] LNet: Added LNI 192.168.202.106@tcp104 [8/256/0/180] [15414.670196] LNet: Added LNI 192.168.202.106@tcp105 [8/256/0/180] [15415.476045] LNet: Added LNI 192.168.202.106@tcp106 [8/256/0/180] [15416.255972] LNet: Added LNI 192.168.202.106@tcp107 [8/256/0/180] [15417.075313] LNet: Added LNI 192.168.202.106@tcp108 [8/256/0/180] [15417.945550] LNet: Added LNI 192.168.202.106@tcp109 [8/256/0/180] [15418.974868] LNet: Added LNI 192.168.202.106@tcp110 [8/256/0/180] [15419.824944] LNet: Added LNI 192.168.202.106@tcp111 [8/256/0/180] [15420.738564] LNet: Added LNI 192.168.202.106@tcp112 [8/256/0/180] [15421.549528] LNet: Added LNI 192.168.202.106@tcp113 [8/256/0/180] [15422.487613] LNet: Added LNI 192.168.202.106@tcp114 [8/256/0/180] [15423.422554] LNet: Added LNI 192.168.202.106@tcp115 [8/256/0/180] [15424.445608] LNet: Added LNI 192.168.202.106@tcp116 [8/256/0/180] [15425.328168] LNet: Added LNI 192.168.202.106@tcp117 [8/256/0/180] [15426.205753] LNet: Added LNI 192.168.202.106@tcp118 [8/256/0/180] [15427.103415] LNet: Added LNI 192.168.202.106@tcp119 [8/256/0/180] [15428.097046] LNet: Added LNI 192.168.202.106@tcp120 [8/256/0/180] [15429.226504] LNet: Added LNI 192.168.202.106@tcp121 [8/256/0/180] [15430.229221] LNet: Added LNI 192.168.202.106@tcp122 [8/256/0/180] [15431.254239] LNet: Added LNI 192.168.202.106@tcp123 [8/256/0/180] [15432.236799] LNet: Added LNI 192.168.202.106@tcp124 [8/256/0/180] [15432.988839] LNet: Added LNI 192.168.202.106@tcp125 [8/256/0/180] [15433.904352] LNet: Added LNI 192.168.202.106@tcp126 [8/256/0/180] [15434.733540] LNet: Added LNI 192.168.202.106@tcp127 [8/256/0/180] [15435.608140] LNet: Added LNI 192.168.202.106@tcp128 [8/256/0/180] [15436.525506] LNet: Added LNI 192.168.202.106@tcp129 [8/256/0/180] [15437.436821] LNet: Added LNI 192.168.202.106@tcp130 [8/256/0/180] [15438.248640] LNet: Added LNI 192.168.202.106@tcp131 [8/256/0/180] [15438.935213] LNet: Added LNI 192.168.202.106@tcp132 [8/256/0/180] [15440.087807] LNet: Added LNI 192.168.202.106@tcp133 [8/256/0/180] [15441.075110] LNet: Added LNI 192.168.202.106@tcp134 [8/256/0/180] [15442.199436] LNet: Added LNI 192.168.202.106@tcp135 [8/256/0/180] [15451.175206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15451.194481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15452.362604] Lustre: 464539:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15452.433692] LNetError: 461996:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15452.457066] LNetError: Protocol error connecting to 192.168.202.106@tcp120 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15452.480727] LNetError: Skipped 5 previous similar messages [15452.651219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15452.710874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15452.850105] Lustre: lustre-MDT0000: new disk, initializing [15453.002602] Lustre: 464597:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15453.015994] Lustre: 464597:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15453.050900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15453.088082] LNet: 6 local NIs in recovery (showing 6): 192.168.202.106@tcp125, 192.168.202.106@tcp126, 192.168.202.106@tcp127, 192.168.202.106@tcp129, 192.168.202.106@tcp120, 192.168.202.106@tcp128 [15453.093353] LNetError: 464569:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15453.134315] LNetError: 464569:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [15453.152705] LNetError: Protocol error connecting to 192.168.202.106@tcp106 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15453.159351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15453.169473] LNetError: Skipped 4 previous similar messages [15459.673963] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15472.703592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15472.903482] Lustre: 464557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15473.025716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15473.036593] Lustre: Skipped 1 previous similar message [15473.150075] Lustre: lustre-MDT0001: new disk, initializing [15473.201602] Lustre: 465359:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15473.219032] Lustre: 465359:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15473.355740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15473.432925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15473.442585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15478.071565] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15479.857235] Lustre: 465711:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15479.867250] Lustre: 465711:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15482.794495] LNetError: 461997:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15482.802345] LNetError: Protocol error connecting to 192.168.202.106@tcp121 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15484.833904] LNetError: 464560:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15484.852485] LNetError: Protocol error connecting to 192.168.202.106@tcp124 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15487.560630] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15489.976365] LNetError: 464561:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15489.990951] LNetError: 464561:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [15490.002203] LNetError: Protocol error connecting to 192.168.202.106@tcp112 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15490.014369] LNetError: Skipped 2 previous similar messages [15494.085583] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15498.214973] LNetError: 461999:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15498.228648] LNetError: 461999:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [15498.236389] LNetError: Protocol error connecting to 192.168.202.106@tcp106 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15498.254231] LNetError: Skipped 3 previous similar messages [15500.884728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15501.278528] Lustre: lustre-OST0000: new disk, initializing [15501.284325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15501.291490] Lustre: 466533:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15501.374372] Lustre: 466539:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.106@tcp: can't add NID 192.168.202.106@tcp130: rc = -75 [15501.385790] Lustre: 466539:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15501.446297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15506.538647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15506.565597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15506.710726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15508.289610] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15515.618732] LNetError: 461996:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15515.628967] LNetError: 461996:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [15515.636996] LNetError: Protocol error connecting to 192.168.202.106@tcp112 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15515.645541] LNetError: Skipped 7 previous similar messages [15519.451745] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15527.393465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15527.403340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15527.412895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15527.416618] Lustre: Skipped 1 previous similar message [15527.437922] Lustre: Skipped 1 previous similar message [15532.513883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15532.520438] Lustre: Skipped 1 previous similar message [15533.723631] Lustre: server umount lustre-OST0000 complete [15540.194288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15540.200697] LustreError: Skipped 1 previous similar message [15540.227537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15540.260372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15542.753116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15542.753565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15544.218294] Lustre: server umount lustre-MDT0000 complete [15548.200169] LustreError: 464549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786704621 with bad export cookie 2140306148443339761 [15548.208609] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15548.215429] LustreError: 464549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15548.388572] LNetError: 464564:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.106 [15548.401542] LNetError: 464564:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15548.417707] LNetError: Protocol error connecting to 192.168.202.106@tcp124 at host 192.168.202.106:988: is it running a compatible version of Lustre? [15548.431298] LNetError: Skipped 8 previous similar messages [15548.611707] Lustre: server umount lustre-MDT0001 complete [15560.611072] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15563.940634] Key type lgssc unregistered [15564.456131] LNet: 468450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15564.470790] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15564.481967] LNet: Removed LNI 192.168.202.106@tcp [15565.537243] LNet: Removed LNI 192.168.202.106@tcp100 [15565.540124] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15565.558174] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15565.574190] LNet: Removed LNI 192.168.202.106@tcp101 [15566.625412] LNet: Removed LNI 192.168.202.106@tcp102 [15566.628292] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15566.634658] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15566.653225] LNet: Removed LNI 192.168.202.106@tcp103 [15567.711226] LNet: Removed LNI 192.168.202.106@tcp104 [15568.737961] LNet: Removed LNI 192.168.202.106@tcp105 [15568.742399] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15568.754815] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15568.772644] LNet: Removed LNI 192.168.202.106@tcp106 [15569.825187] LNet: Removed LNI 192.168.202.106@tcp107 [15570.847277] LNet: Removed LNI 192.168.202.106@tcp108 [15571.871521] LNet: Removed LNI 192.168.202.106@tcp109 [15572.895157] LNet: Removed LNI 192.168.202.106@tcp110 [15572.899555] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15572.905958] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15572.911356] LNet: Removed LNI 192.168.202.106@tcp111 [15573.919190] LNet: Removed LNI 192.168.202.106@tcp112 [15574.943733] LNet: Removed LNI 192.168.202.106@tcp113 [15575.968067] LNet: Removed LNI 192.168.202.106@tcp114 [15576.991219] LNet: Removed LNI 192.168.202.106@tcp115 [15578.015410] LNet: Removed LNI 192.168.202.106@tcp116 [15579.043370] LNet: Removed LNI 192.168.202.106@tcp117 [15580.063218] LNet: Removed LNI 192.168.202.106@tcp118 [15581.087274] LNet: Removed LNI 192.168.202.106@tcp119 [15581.095197] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15581.108814] LNetError: 468450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15581.125049] LNet: Removed LNI 192.168.202.106@tcp120 [15582.176728] LNet: Removed LNI 192.168.202.106@tcp121 [15583.199945] LNet: Removed LNI 192.168.202.106@tcp122 [15584.223679] LNet: Removed LNI 192.168.202.106@tcp123 [15585.247216] LNet: Removed LNI 192.168.202.106@tcp124 [15586.271379] LNet: Removed LNI 192.168.202.106@tcp125 [15587.297402] LNet: Removed LNI 192.168.202.106@tcp126 [15588.383873] LNet: Removed LNI 192.168.202.106@tcp127 [15589.408204] LNet: Removed LNI 192.168.202.106@tcp128 [15590.432738] LNet: Removed LNI 192.168.202.106@tcp129 [15590.436696] LNet: Removed LNI 192.168.202.106@tcp130 [15590.440549] LNet: Removed LNI 192.168.202.106@tcp131 [15590.448408] LNet: Removed LNI 192.168.202.106@tcp132 [15590.450480] LNet: Removed LNI 192.168.202.106@tcp133 [15590.461640] LNet: Removed LNI 192.168.202.106@tcp134 [15590.505205] LNet: Removed LNI 192.168.202.106@tcp135 [15591.553625] Key type .llcrypt unregistered [15591.556082] Key type ._llcrypt unregistered [15620.290919] Key type ._llcrypt registered [15620.294580] Key type .llcrypt registered [15620.424900] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [15638.061723] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15639.589986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15639.622096] alg: No test for adler32 (adler32-zlib) [15640.623237] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [15640.832841] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15642.505505] Key type lgssc registered [15643.845407] Lustre: Echo OBD driver; http://www.lustre.org/ [15689.825800] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:52:41 (1786704761) [15712.425393] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [15716.139806] Key type lgssc unregistered [15716.436078] LNet: 472557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15716.442157] LNetError: 472557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15716.470640] LNet: Removed LNI 192.168.202.106@tcp [15717.353609] Key type .llcrypt unregistered [15717.355417] Key type ._llcrypt unregistered [15736.146177] Key type ._llcrypt registered [15736.147758] Key type .llcrypt registered [15736.245985] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [15737.466724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15737.600547] alg: No test for adler32 (adler32-zlib) [15738.733175] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [15739.039590] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [15740.831219] Key type lgssc registered [15741.842834] Lustre: Echo OBD driver; http://www.lustre.org/ [15753.719986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15753.755225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15755.181449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15755.257565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15755.351840] Lustre: lustre-MDT0000: new disk, initializing [15755.516229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15755.572694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15761.608364] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15774.275584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15774.485281] Lustre: 473772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15774.557823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15774.561561] Lustre: Skipped 1 previous similar message [15774.688311] Lustre: lustre-MDT0001: new disk, initializing [15774.838335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15774.883222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15774.900540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15779.487098] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15789.359159] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15796.086210] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15802.590981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15802.893966] Lustre: lustre-OST0000: new disk, initializing [15802.898687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15802.904602] Lustre: 475742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15802.996491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15812.090572] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [15812.234094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15812.275378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15812.475395] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15824.224413] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15828.473956] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing lnet_if_list [15834.976522] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15834.995529] LNet: Added LNI 192.168.202.106@tcp10 [8/256/0/180] [15841.327780] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15841.358029] LNet: Added LNI 192.168.202.106@tcp11 [8/256/0/180] [15847.305810] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15847.339332] LNet: Added LNI 192.168.202.106@tcp12 [8/256/0/180] [15853.957749] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15854.028088] LNet: Added LNI 192.168.202.106@tcp13 [8/256/0/180] [15861.301509] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15861.379726] LNet: Added LNI 192.168.202.106@tcp14 [8/256/0/180] [15868.774397] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15868.808080] LNet: Added LNI 192.168.202.106@tcp15 [8/256/0/180] [15876.358839] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15876.428295] LNet: Added LNI 192.168.202.106@tcp16 [8/256/0/180] [15883.193860] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15883.214322] LNet: Added LNI 192.168.202.106@tcp17 [8/256/0/180] [15890.133459] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15890.206349] LNet: Added LNI 192.168.202.106@tcp18 [8/256/0/180] [15897.286939] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15897.342858] LNet: Added LNI 192.168.202.106@tcp19 [8/256/0/180] [15903.333727] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15903.374247] LNet: Added LNI 192.168.202.106@tcp20 [8/256/0/180] [15908.867625] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15908.896453] LNet: Added LNI 192.168.202.106@tcp21 [8/256/0/180] [15914.455512] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15914.498630] LNet: Added LNI 192.168.202.106@tcp22 [8/256/0/180] [15919.762742] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15919.783547] LNet: Added LNI 192.168.202.106@tcp23 [8/256/0/180] [15925.529718] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15925.549566] LNet: Added LNI 192.168.202.106@tcp24 [8/256/0/180] [15930.689567] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15930.717055] LNet: Added LNI 192.168.202.106@tcp25 [8/256/0/180] [15936.138108] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15936.165994] LNet: Added LNI 192.168.202.106@tcp26 [8/256/0/180] [15941.813230] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15941.829503] LNet: Added LNI 192.168.202.106@tcp27 [8/256/0/180] [15947.905955] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15947.940197] LNet: Added LNI 192.168.202.106@tcp28 [8/256/0/180] [15954.901384] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15954.936894] LNet: Added LNI 192.168.202.106@tcp29 [8/256/0/180] [15961.702147] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15961.726335] LNet: Added LNI 192.168.202.106@tcp30 [8/256/0/180] [15968.892947] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15968.921368] LNet: Added LNI 192.168.202.106@tcp31 [8/256/0/180] [15976.254172] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15976.315638] LNet: Added LNI 192.168.202.106@tcp32 [8/256/0/180] [15984.210517] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15984.264521] LNet: Added LNI 192.168.202.106@tcp33 [8/256/0/180] [15991.873437] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15991.903622] LNet: Added LNI 192.168.202.106@tcp34 [8/256/0/180] [15998.595893] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15998.624742] LNet: Added LNI 192.168.202.106@tcp35 [8/256/0/180] [16005.999923] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16006.037537] LNet: Added LNI 192.168.202.106@tcp36 [8/256/0/180] [16013.507221] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16013.520887] LNet: Added LNI 192.168.202.106@tcp37 [8/256/0/180] [16021.740763] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16021.777956] LNet: Added LNI 192.168.202.106@tcp38 [8/256/0/180] [16029.721105] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16029.761162] LNet: Added LNI 192.168.202.106@tcp39 [8/256/0/180] [16036.858370] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16036.878422] LNet: Added LNI 192.168.202.106@tcp40 [8/256/0/180] [16044.222500] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16044.280631] LNet: Added LNI 192.168.202.106@tcp41 [8/256/0/180] [16052.544931] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16052.572366] LNet: Added LNI 192.168.202.106@tcp42 [8/256/0/180] [16060.291968] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16060.335331] LNet: Added LNI 192.168.202.106@tcp43 [8/256/0/180] [16068.523421] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16068.569337] LNet: Added LNI 192.168.202.106@tcp44 [8/256/0/180] [16076.557560] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16076.593842] LNet: Added LNI 192.168.202.106@tcp45 [8/256/0/180] [16084.361431] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16084.390802] LNet: Added LNI 192.168.202.106@tcp46 [8/256/0/180] [16093.055662] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16093.086772] LNet: Added LNI 192.168.202.106@tcp47 [8/256/0/180] [16101.252568] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16101.295237] LNet: Added LNI 192.168.202.106@tcp48 [8/256/0/180] [16110.175857] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16110.201504] LNet: Added LNI 192.168.202.106@tcp49 [8/256/0/180] [16120.517719] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16120.535982] LNet: Added LNI 192.168.202.106@tcp50 [8/256/0/180] [16148.448530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16148.458458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16148.483341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16150.505377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16150.513894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16150.537385] Lustre: Skipped 2 previous similar messages [16155.617951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16155.624307] Lustre: Skipped 1 previous similar message [16160.223278] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16160.475657] Lustre: server umount lustre-MDT0000 complete [16165.064628] LustreError: 473763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786705238 with bad export cookie 11416832765507627756 [16165.072911] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16165.073352] LustreError: 473763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16165.644721] Lustre: server umount lustre-MDT0001 complete [16181.215181] Lustre: 473044:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786705238/real 1786705238] req@0000000021af7dbf x1873495785133440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786705254 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16181.238536] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16181.252716] Lustre: Skipped 1 previous similar message [16187.365441] Lustre: 473043:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786705244/real 1786705244] req@000000000c42f2c7 x1873495785133568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786705260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16187.570905] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16191.456267] Lustre: 473042:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786705248/real 1786705248] req@000000005e6f769d x1873495785133824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786705264 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16196.576219] Lustre: 473044:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786705253/real 1786705253] req@0000000050cfb288 x1873495785133952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786705269 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16196.836215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16210.913845] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9e70bce9481f755e [16210.939324] Lustre: MGC192.168.202.106@tcp: Connection restored to 0@lo (at 0@lo) [16211.192522] LustreError: 475736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16211.204842] LustreError: 475736:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16211.282497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16215.399554] LustreError: 475737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16215.414701] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16216.435535] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16220.647415] LustreError: 475736:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16225.768878] LustreError: 475737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16225.987658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16226.430564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16227.450741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16231.706682] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16240.846672] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16247.206951] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16272.362591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16272.371464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16272.390594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16276.892156] Lustre: server umount lustre-OST0000 complete [16282.594108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16282.597039] LustreError: Skipped 1 previous similar message [16282.599470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16282.606068] Lustre: Skipped 1 previous similar message [16282.611849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16282.620292] Lustre: Skipped 3 previous similar messages [16294.368541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16294.377656] Lustre: Skipped 5 previous similar messages [16295.391140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16295.525234] Lustre: server umount lustre-MDT0000 complete [16299.082926] LustreError: 474569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786705372 with bad export cookie 11416832765507630430 [16299.084664] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16299.091636] LustreError: 474569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16299.464845] Lustre: server umount lustre-MDT0001 complete [16311.222377] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16313.559412] Key type lgssc unregistered [16313.876666] LNet: 502819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16313.885034] LNetError: 502819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16314.911352] LNet: Removed LNI 192.168.202.106@tcp [16314.914342] LNetError: 502819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16314.925637] LNet: Removed LNI 192.168.202.106@tcp10 [16314.927122] LNet: Removed LNI 192.168.202.106@tcp11 [16314.937976] LNet: Removed LNI 192.168.202.106@tcp12 [16314.939952] LNet: Removed LNI 192.168.202.106@tcp13 [16314.948204] LNet: Removed LNI 192.168.202.106@tcp14 [16314.952994] LNet: Removed LNI 192.168.202.106@tcp15 [16314.957109] LNet: Removed LNI 192.168.202.106@tcp16 [16314.958540] LNet: Removed LNI 192.168.202.106@tcp17 [16314.962015] LNet: Removed LNI 192.168.202.106@tcp18 [16314.965324] LNet: Removed LNI 192.168.202.106@tcp19 [16314.967736] LNet: Removed LNI 192.168.202.106@tcp20 [16314.972707] LNet: Removed LNI 192.168.202.106@tcp21 [16314.974465] LNet: Removed LNI 192.168.202.106@tcp22 [16314.975880] LNet: Removed LNI 192.168.202.106@tcp23 [16314.977239] LNet: Removed LNI 192.168.202.106@tcp24 [16314.982175] LNet: Removed LNI 192.168.202.106@tcp25 [16314.985513] LNet: Removed LNI 192.168.202.106@tcp26 [16314.991550] LNet: Removed LNI 192.168.202.106@tcp27 [16314.994420] LNet: Removed LNI 192.168.202.106@tcp28 [16315.000594] LNet: Removed LNI 192.168.202.106@tcp29 [16315.005724] LNet: Removed LNI 192.168.202.106@tcp30 [16315.012267] LNet: Removed LNI 192.168.202.106@tcp31 [16315.013547] LNet: Removed LNI 192.168.202.106@tcp32 [16315.014667] LNet: Removed LNI 192.168.202.106@tcp33 [16315.018346] LNet: Removed LNI 192.168.202.106@tcp34 [16315.021440] LNet: Removed LNI 192.168.202.106@tcp35 [16315.023422] LNet: Removed LNI 192.168.202.106@tcp36 [16315.028935] LNet: Removed LNI 192.168.202.106@tcp37 [16315.036850] LNet: Removed LNI 192.168.202.106@tcp38 [16315.041548] LNet: Removed LNI 192.168.202.106@tcp39 [16315.047375] LNet: Removed LNI 192.168.202.106@tcp40 [16315.051418] LNet: Removed LNI 192.168.202.106@tcp41 [16315.054141] LNet: Removed LNI 192.168.202.106@tcp42 [16315.056179] LNet: Removed LNI 192.168.202.106@tcp43 [16315.062323] LNet: Removed LNI 192.168.202.106@tcp44 [16315.064729] LNet: Removed LNI 192.168.202.106@tcp45 [16315.066830] LNet: Removed LNI 192.168.202.106@tcp46 [16315.069089] LNet: Removed LNI 192.168.202.106@tcp47 [16315.071695] LNet: Removed LNI 192.168.202.106@tcp48 [16315.074129] LNet: Removed LNI 192.168.202.106@tcp49 [16315.082778] LNet: Removed LNI 192.168.202.106@tcp50 [16315.737946] Key type .llcrypt unregistered [16315.739894] Key type ._llcrypt unregistered [16334.803302] Key type ._llcrypt registered [16334.807166] Key type .llcrypt registered [16334.906817] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [16350.072606] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16351.193672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16351.244491] alg: No test for adler32 (adler32-zlib) [16352.318473] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [16352.602335] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16354.271217] Key type lgssc registered [16355.564186] Lustre: Echo OBD driver; http://www.lustre.org/ [16394.677280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16394.724323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16396.237879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16396.369709] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16396.501473] Lustre: lustre-MDT0000: new disk, initializing [16396.610895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16396.636181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16401.672178] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16413.663208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16413.783839] Lustre: 506015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16413.832607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16413.836236] Lustre: Skipped 1 previous similar message [16413.916352] Lustre: lustre-MDT0001: new disk, initializing [16414.032466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16414.086709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16414.098936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16418.978533] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16429.533803] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16436.363294] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16444.815548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16445.157622] Lustre: *** cfs_fail_loc=90f, val=0*** [16445.268275] Lustre: lustre-OST0000: new disk, initializing [16445.275735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16445.287246] Lustre: 508032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16445.407546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16451.123501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16451.148609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16451.301535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16453.235805] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16464.060834] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16477.153105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16477.168050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16477.178891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16477.196330] Lustre: Skipped 1 previous similar message [16477.226126] Lustre: Skipped 1 previous similar message [16478.950592] Lustre: server umount lustre-OST0000 complete [16485.865242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16485.875735] LustreError: Skipped 1 previous similar message [16485.878413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16485.892772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16487.395244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16487.395263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16489.896065] Lustre: server umount lustre-MDT0000 complete [16494.327171] LustreError: 506008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786705567 with bad export cookie 2003079955080871386 [16494.336171] LustreError: 506008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16494.336502] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16494.703656] Lustre: server umount lustre-MDT0001 complete [16509.866833] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16513.727831] Key type lgssc unregistered [16514.091239] LNet: 509942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16514.099495] LNetError: 509942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16515.177557] LNet: Removed LNI 192.168.202.106@tcp [16516.177797] Key type .llcrypt unregistered [16516.187654] Key type ._llcrypt unregistered [16540.304469] Key type ._llcrypt registered [16540.306460] Key type .llcrypt registered [16540.393676] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16541.809623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16541.855658] alg: No test for adler32 (adler32-zlib) [16542.895833] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [16543.114559] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16544.808791] Key type lgssc registered [16546.073099] Lustre: Echo OBD driver; http://www.lustre.org/ [16555.110408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16555.178400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16556.980912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16563.350719] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16567.277663] LustreError: 511216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16567.302169] LustreError: 511216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16572.408690] LustreError: 511215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16573.700609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16574.333107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16579.496113] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16588.853465] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16596.822913] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16603.828758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16604.207229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16610.316462] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16618.835391] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16641.070697] loop0: detected capacity change from 0 to 204800000 [16641.322695] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16641.340805] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16641.359602] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16648.475696] loop0: detected capacity change from 0 to 204800000 [16648.740404] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16648.755914] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16648.788148] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16654.967573] loop0: detected capacity change from 0 to 204800000 [16662.088658] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16662.323244] Lustre: fs15246-OST0000: new disk, initializing [16662.336407] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16662.344496] Lustre: 514834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16662.464650] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16670.605412] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16681.944568] loop1: detected capacity change from 0 to 204800000 [16688.261830] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16688.397837] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16688.445386] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16688.551284] Lustre: fs15246-MDT0000: new disk, initializing [16688.783558] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16688.811449] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16688.918211] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16688.944897] Lustre: 515956:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000ea2c6ca x1873496628056192/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786705778 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16693.971322] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16694.058911] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16694.068660] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16694.134672] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16719.349852] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16719.372816] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16724.226086] Lustre: server umount fs15246-MDT0000 complete [16724.490484] LustreError: 511212:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16729.590104] LustreError: 512313:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16734.025266] Lustre: server umount fs15246-OST0000 complete [16747.492663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16747.498393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16747.509109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16750.052064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16750.059457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16750.062578] Lustre: Skipped 1 previous similar message [16753.778926] Lustre: server umount lustre-OST0000 complete [16758.753031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16758.767143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16758.783846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16764.344524] Lustre: server umount lustre-MDT0000 complete [16765.410606] LustreError: 511216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16765.451736] LustreError: 511216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16768.948349] LustreError: 513061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786705842 with bad export cookie 3342204209084395156 [16768.965977] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16768.967234] LustreError: 513061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16769.455747] Lustre: server umount lustre-MDT0001 complete [16784.953604] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [16788.620729] Key type lgssc unregistered [16788.998620] LNet: 519059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16789.007614] LNetError: 519059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16789.025876] LNet: Removed LNI 192.168.202.106@tcp [16790.116565] Key type .llcrypt unregistered [16790.121860] Key type ._llcrypt unregistered [16812.999886] Key type ._llcrypt registered [16813.005855] Key type .llcrypt registered [16813.177752] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16814.402744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16814.441017] alg: No test for adler32 (adler32-zlib) [16815.742589] Lustre: Lustre: Build Version: 2.17.57_1_g9a8e296 [16815.970185] LNet: Added LNI 192.168.202.106@tcp [8/256/0/180] [16817.655192] Key type lgssc registered [16819.160677] Lustre: Echo OBD driver; http://www.lustre.org/ [16864.249642] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [16876.319560] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [16920.474251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16920.588246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16921.982950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16922.014235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16922.161835] Lustre: lustre-MDT0000: new disk, initializing [16922.350192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16922.370944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16927.298749] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16938.606508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16938.755486] Lustre: 523530:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16938.814489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16938.819513] Lustre: Skipped 1 previous similar message [16938.895584] Lustre: lustre-MDT0001: new disk, initializing [16938.979054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16939.005306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16939.014758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16943.421672] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16954.132158] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16961.295471] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16970.746812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16971.136492] Lustre: lustre-OST0000: new disk, initializing [16971.144654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16971.159465] Lustre: 525500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16971.341626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16977.507701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16977.523158] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16977.653449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16979.052100] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_default_debug -1 all [16991.280972] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16997.190256] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16997.400758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17003.930904] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17004.183346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17008.610660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17008.623826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17008.644670] Lustre: Skipped 1 previous similar message [17008.647447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17008.662033] Lustre: Skipped 1 previous similar message [17013.742529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17013.756335] Lustre: Skipped 1 previous similar message [17018.854430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17020.386043] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17020.591836] Lustre: server umount lustre-OST0000 complete [17026.016735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17026.030540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17026.043416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17026.048143] Lustre: Skipped 1 previous similar message [17029.090628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17031.003520] Lustre: server umount lustre-MDT0000 complete [17035.562760] LustreError: 526840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786706108 with bad export cookie 15067609634542252275 [17035.566423] LustreError: MGC192.168.202.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17035.583684] LustreError: 526840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17036.053869] Lustre: server umount lustre-MDT0001 complete [17063.719124] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing set_hostid [17076.316435] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing load_modules_local [17114.493212] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16867 sec ========= 07:16:26 (1786706186) [17116.267472] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:16:27 (1786706187) === [17119.700562] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:16:31 (1786706191) === [17160.398555] Lustre: DEBUG MARKER: oleg206-server.virtnet: executing unload_modules_local [17163.217749] Key type lgssc unregistered [17163.525969] LNet: 532151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17163.536839] LNetError: 532151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17163.552924] LNet: Removed LNI 192.168.202.106@tcp [17164.510650] Key type .llcrypt unregistered [17164.513065] Key type ._llcrypt unregistered