[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 524443580 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002374] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007261] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.011041] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014082] *** VALIDATE selinux *** [ 0.023169] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028272] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] *** VALIDATE tmpfs *** [ 0.032500] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039009] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043299] debug: unmapping init [mem 0xffffffffa6259000-0xffffffffa6260fff] [ 0.045278] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046827] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047030] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054302] rcu: Hierarchical SRCU implementation. [ 0.056706] smp: Bringing up secondary CPUs ... [ 0.057676] x86: Booting SMP configuration: [ 0.058027] .... node #0, CPUs: #1 #2 #3 [ 0.062203] smp: Brought up 1 node, 4 CPUs [ 0.064015] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.136796] node 0 deferred pages initialised in 69ms [ 0.140039] devtmpfs: initialized [ 0.141189] x86/mm: Memory block size: 128MB [ 0.144031] gcov: version magic: 0x41383552 [ 0.146296] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148247] pinctrl core: initialized pinctrl subsystem [ 0.149212] [ 0.149911] ************************************************************* [ 0.150013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151016] ** ** [ 0.152014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153017] ** ** [ 0.154015] ** This means that this kernel is built to expose internal ** [ 0.155016] ** IOMMU data structures, which may compromise security on ** [ 0.156011] ** your system. ** [ 0.157015] ** ** [ 0.158013] ** If you see this message and you are not debugging the ** [ 0.159018] ** kernel, report this immediately to your vendor! ** [ 0.160013] ** ** [ 0.161017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162077] ************************************************************* [ 0.163856] NET: Registered protocol family 16 [ 0.164461] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168037] cpuidle: using governor menu [ 0.169876] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171438] PCI: Using configuration type 1 for base access [ 0.173129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183049] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.185158] cryptd: max_cpu_qlen set to 1000 [ 0.187293] ACPI: Added _OSI(Module Device) [ 0.188013] ACPI: Added _OSI(Processor Device) [ 0.189013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194189] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.196377] ACPI: Interpreter enabled [ 0.197045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.198016] ACPI: Using IOAPIC for interrupt routing [ 0.199119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.200543] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211220] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.212043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.213021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216438] acpiphp: Slot [2] registered [ 0.217143] acpiphp: Slot [5] registered [ 0.218151] acpiphp: Slot [6] registered [ 0.219128] acpiphp: Slot [7] registered [ 0.220118] acpiphp: Slot [8] registered [ 0.221100] acpiphp: Slot [9] registered [ 0.222143] acpiphp: Slot [10] registered [ 0.223331] acpiphp: Slot [3] registered [ 0.224097] acpiphp: Slot [4] registered [ 0.225251] acpiphp: Slot [11] registered [ 0.226090] acpiphp: Slot [12] registered [ 0.227262] acpiphp: Slot [13] registered [ 0.228081] acpiphp: Slot [14] registered [ 0.229090] acpiphp: Slot [15] registered [ 0.230115] acpiphp: Slot [16] registered [ 0.231116] acpiphp: Slot [17] registered [ 0.232120] acpiphp: Slot [18] registered [ 0.233120] acpiphp: Slot [19] registered [ 0.234107] acpiphp: Slot [20] registered [ 0.235137] acpiphp: Slot [21] registered [ 0.236134] acpiphp: Slot [22] registered [ 0.237114] acpiphp: Slot [23] registered [ 0.238169] acpiphp: Slot [24] registered [ 0.239146] acpiphp: Slot [25] registered [ 0.241101] acpiphp: Slot [26] registered [ 0.242112] acpiphp: Slot [27] registered [ 0.244118] acpiphp: Slot [28] registered [ 0.246119] acpiphp: Slot [29] registered [ 0.247114] acpiphp: Slot [30] registered [ 0.248093] acpiphp: Slot [31] registered [ 0.249089] PCI host bridge to bus 0000:00 [ 0.251018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.258023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.260028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263058] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.265221] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.269095] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.272537] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.284956] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.289039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.294020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.297018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.301465] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.304811] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.307034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.309797] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.328015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.333016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.339878] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.348015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.356015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.390358] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.399016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.408015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.432017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439138] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.447018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.457016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.491020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.508399] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.523016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.542017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.556553] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.563014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.573020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.595018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.607719] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.617018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.635027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.645644] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.647512] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.650381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652422] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.659104] iommu: Default domain type: Passthrough [ 0.662623] SCSI subsystem initialized [ 0.664144] ACPI: bus type USB registered [ 0.665215] usbcore: registered new interface driver usbfs [ 0.667095] usbcore: registered new interface driver hub [ 0.669109] usbcore: registered new device driver usb [ 0.671201] pps_core: LinuxPPS API ver. 1 registered [ 0.673013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.676072] PTP clock support registered [ 0.678077] EDAC MC: Ver: 3.0.0 [ 0.680107] PCI: Using ACPI for IRQ routing [ 0.681916] NetLabel: Initializing [ 0.683012] NetLabel: domain hash size = 128 [ 0.685012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.687089] NetLabel: unlabeled traffic allowed by default [ 0.689405] vgaarb: loaded [ 0.691301] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.699000] clocksource: Switched to clocksource kvm-clock [ 0.803962] VFS: Disk quotas dquot_6.6.0 [ 0.805414] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.807649] *** VALIDATE ramfs *** [ 0.808766] *** VALIDATE hugetlbfs *** [ 0.810351] pnp: PnP ACPI init [ 0.813422] pnp: PnP ACPI: found 6 devices [ 0.835421] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838376] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.840292] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.842442] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.844387] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.846771] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.849813] NET: Registered protocol family 2 [ 0.852302] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.857941] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.862183] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867323] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.871143] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874445] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877854] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.881171] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.884729] NET: Registered protocol family 1 [ 0.888487] RPC: Registered named UNIX socket transport module. [ 0.890745] RPC: Registered udp transport module. [ 0.892311] RPC: Registered tcp transport module. [ 0.894375] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.896899] NET: Registered protocol family 44 [ 0.898085] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.899818] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901329] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.903291] PCI: CLS 0 bytes, default 64 [ 0.904924] Unpacking initramfs... [ 2.307501] debug: unmapping init [mem 0xffff89c9fcc54000-0xffff89c9fffbffff] [ 2.311265] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.313564] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.316692] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.826125] Initialise system trusted keyrings [ 2.828065] Key type blacklist registered [ 2.830645] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.839517] zbud: loaded [ 2.842806] *** VALIDATE nfs *** [ 2.844449] *** VALIDATE nfs4 *** [ 2.846564] pstore: using deflate compression [ 2.850181] Platform Keyring initialized [ 2.955232] NET: Registered protocol family 38 [ 2.957088] Key type asymmetric registered [ 2.958228] Asymmetric key parser 'x509' registered [ 2.959425] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.962674] io scheduler mq-deadline registered [ 2.964220] io scheduler kyber registered [ 2.965731] io scheduler bfq registered [ 2.967564] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.970482] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.973413] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.976310] ACPI: Power Button [PWRF] [ 2.981219] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.988189] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.002377] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.009719] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.022731] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.050208] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.078918] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.083894] Non-volatile memory driver v1.3 [ 3.085671] Linux agpgart interface v0.103 [ 3.115757] virtio_blk virtio1: [vda] 149424 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.118573] vda: detected capacity change from 0 to 76505088 [ 3.133964] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.137349] vdb: detected capacity change from 0 to 1073741824 [ 3.151536] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.155336] vdc: detected capacity change from 0 to 2621440000 [ 3.173757] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.176768] vdd: detected capacity change from 0 to 2621440000 [ 3.189310] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.192182] vde: detected capacity change from 0 to 4294967296 [ 3.206806] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.209588] vdf: detected capacity change from 0 to 4294967296 [ 3.215650] libphy: Fixed MDIO Bus: probed [ 3.222658] usbcore: registered new interface driver usbserial_generic [ 3.225371] usbserial: USB Serial support registered for generic [ 3.227586] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.233266] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.235151] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.237478] mousedev: PS/2 mouse device common for all mice [ 3.240583] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.242677] rtc_cmos 00:05: RTC can wake from S4 [ 3.246824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.247135] rtc_cmos 00:05: registered as rtc0 [ 3.250631] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.253062] intel_pstate: CPU model not supported [ 3.254927] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.258266] hid: raw HID events driver (C) Jiri Kosina [ 3.261040] usbcore: registered new interface driver usbhid [ 3.264344] usbhid: USB HID core driver [ 3.266188] drop_monitor: Initializing network drop monitor service [ 3.269440] Initializing XFRM netlink socket [ 3.271703] NET: Registered protocol family 10 [ 3.275216] Segment Routing with IPv6 [ 3.276961] NET: Registered protocol family 17 [ 3.278529] mpls_gso: MPLS GSO support [ 3.284657] RAS: Correctable Errors collector initialized. [ 3.286570] AVX version of gcm_enc/dec engaged. [ 3.288015] AES CTR mode by8 optimization enabled [ 3.370874] sched_clock: Marking stable (3370852111, 0)->(4369283961, -998431850) [ 3.374689] registered taskstats version 1 [ 3.376435] Loading compiled-in X.509 certificates [ 3.378063] zswap: loaded using pool lzo/zbud [ 3.403962] Key type big_key registered [ 3.416426] Key type encrypted registered [ 3.418021] ima: No TPM chip found, activating TPM-bypass! [ 3.419948] ima: Allocated hash algorithm: sha1 [ 3.421452] ima: No architecture policies found [ 3.423045] evm: Initialising EVM extended attributes: [ 3.424640] evm: security.selinux [ 3.425723] evm: security.ima [ 3.426627] evm: security.capability [ 3.427965] evm: HMAC attrs: 0x1 [ 3.430182] rtc_cmos 00:05: setting system clock to 2026-08-22 08:31:21 UTC (1787387481) [ 3.436202] debug: unmapping init [mem 0xffffffffa7203000-0xffffffffa73fffff] [ 3.438906] debug: unmapping init [mem 0xffffffffa5f82000-0xffffffffa6258fff] [ 3.447180] Write protecting the kernel read-only data: 28672k [ 3.450699] debug: unmapping init [mem 0xffffffffa4603000-0xffffffffa47fffff] [ 3.453415] debug: unmapping init [mem 0xffffffffa4f14000-0xffffffffa4ffffff] [ 3.489086] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.496794] systemd[1]: Detected virtualization kvm. [ 3.498477] systemd[1]: Detected architecture x86-64. [ 3.500408] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.527923] systemd[1]: No hostname configured. [ 3.529722] systemd[1]: Set hostname to . [ 3.531663] random: systemd: uninitialized urandom read (16 bytes read) [ 3.534101] systemd[1]: Initializing machine ID from random generator. [ 3.655257] random: systemd: uninitialized urandom read (16 bytes read) [ 3.657895] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.662072] random: systemd: uninitialized urandom read (16 bytes read) [ 3.664525] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.673968] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.390747] device-mapper: uevent: version 1.0.3 [ 4.392662] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.245271] random: fast init done [ 5.273328] virtio_net virtio0 ens2: renamed from eth0 [ 5.277849] scsi host0: ata_piix [ 5.295945] scsi host1: ata_piix [ 5.297417] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.299972] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.691988] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.933210] random: crng init done [ 9.934869] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.450995] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.634790] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.932223] SELinux: Disabled at runtime. [ 11.994200] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.002870] systemd[1]: Detected virtualization kvm. [ 12.004875] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.540303] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.543500] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.548255] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.554279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.558392] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.566057] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.584426] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.693387] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.031121] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.344238] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.366077] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.470090] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.482941] EDAC sbridge: Ver: 1.1.2 [ 15.285939] Key type dns_resolver registered [ 15.607506] NFS: Registering the id_resolver key type [ 15.609398] Key type id_resolver registered [ 15.612236] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 43.810490] libcfs: loading out-of-tree module taints kernel. [ 43.830067] Key type ._llcrypt registered [ 43.831520] Key type .llcrypt registered [ 43.878706] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 53.254702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 53.896038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.906372] alg: No test for adler32 (adler32-zlib) [ 54.941129] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 55.324954] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 56.967469] Key type lgssc registered [ 59.042799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 70.377006] hrtimer: interrupt took 1635291 ns [ 79.325632] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.941709] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 142.275213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.302635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.594215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.622479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.730933] Lustre: lustre-MDT0000: new disk, initializing [ 143.808942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.834134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.980396] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.728926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.867318] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 163.946969] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 163.956242] Lustre: Skipped 1 previous similar message [ 164.136712] Lustre: lustre-MDT0001: new disk, initializing [ 164.258339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.304393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.312351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.048984] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.949221] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.299505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.566442] Lustre: lustre-OST0000: new disk, initializing [ 186.582568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.590746] Lustre: 8434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.716528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.061936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.075880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.262923] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.344685] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.623728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.784356] Lustre: lustre-OST0001: new disk, initializing [ 213.787736] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 213.792940] Lustre: 9508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.869852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.183889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.202200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.348852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.775348] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.617488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.884802] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.762547] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 257.891414] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 263.613340] Lustre: DEBUG MARKER: Client: 2.17.54.111 [ 266.648628] Lustre: DEBUG MARKER: MDS: 2.17.54.111 [ 269.334034] Lustre: DEBUG MARKER: OSS: 2.17.54.111 [ 271.182068] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 04:35:47 EDT 2026 [ 291.707104] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 293.226778] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 311.779275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.783530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 311.800437] Lustre: Skipped 3 previous similar messages [ 311.814140] Lustre: Skipped 2 previous similar messages [ 315.957167] Lustre: server umount lustre-MDT0000 complete [ 320.362970] LustreError: 6487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387798 with bad export cookie 14797419086565740433 [ 320.373941] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.389825] LustreError: 6487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 322.016756] LustreError: 10343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.019679] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.037267] LustreError: 10343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 322.085508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.098529] Lustre: Skipped 1 previous similar message [ 325.025043] LustreError: 10343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.028256] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.040427] LustreError: 10343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 325.047460] Lustre: Skipped 1 previous similar message [ 326.970868] Lustre: server umount lustre-MDT0001 complete [ 331.609661] Lustre: server umount lustre-OST0000 complete [ 335.595573] Lustre: server umount lustre-OST0001 complete [ 343.451181] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 356.561780] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 401.696288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.035904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.057938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.140417] Lustre: lustre-MDT0000: new disk, initializing [ 402.280108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.303962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.783646] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 418.990852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.131870] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.191575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.196665] Lustre: Skipped 1 previous similar message [ 419.306964] Lustre: lustre-MDT0001: new disk, initializing [ 419.426756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 419.508610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 419.525653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 424.446906] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 434.770890] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 440.592124] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 447.184634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 447.459744] Lustre: lustre-OST0000: new disk, initializing [ 447.466268] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 447.479108] Lustre: 17034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 447.569639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.506228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.522108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.632748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.081307] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 463.409122] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 469.008616] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.235769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.267522] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 474.507340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.230554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.237483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.240592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.258619] Lustre: Skipped 2 previous similar messages [ 480.282294] LustreError: Skipped 1 previous similar message [ 482.787345] Lustre: server umount lustre-OST0000 complete [ 490.464849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.486635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.496302] Lustre: Skipped 2 previous similar messages [ 492.887797] Lustre: server umount lustre-MDT0000 complete [ 495.584630] LustreError: 15071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.595234] LustreError: 15071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 496.956422] LustreError: 15861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787387975 with bad export cookie 14797419086565741966 [ 496.968661] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 496.972101] LustreError: 15861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 497.339981] Lustre: server umount lustre-MDT0001 complete [ 501.265582] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 04:39:38 (1787387978) [ 503.436989] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 505.830673] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 04:39:42 (1787387982) [ 514.068325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.712074] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.890274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.170050] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.283221] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 529.386218] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.403723] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.911350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.097282] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 542.866592] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 548.919602] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.104760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 563.019486] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 570.916252] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 585.187340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 585.195195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.211562] Lustre: Skipped 1 previous similar message [ 585.227281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.215151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.040490] Lustre: server umount lustre-OST0000 complete [ 596.451709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.469558] Lustre: Skipped 1 previous similar message [ 598.236855] Lustre: server umount lustre-MDT0000 complete [ 601.569606] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.597426] LustreError: 19445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 602.456708] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388080 with bad export cookie 14797419086565742848 [ 602.464205] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.481173] LustreError: 19425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 602.823120] Lustre: server umount lustre-MDT0001 complete [ 613.936220] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 616.766399] Key type lgssc unregistered [ 617.069931] LNet: 22976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.088047] LNetError: 22976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.117134] LNet: Removed LNI 192.168.204.131@tcp [ 618.398724] Key type .llcrypt unregistered [ 618.404165] Key type ._llcrypt unregistered [ 641.092628] Key type ._llcrypt registered [ 641.094700] Key type .llcrypt registered [ 641.255559] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 642.491874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 642.547459] alg: No test for adler32 (adler32-zlib) [ 643.612661] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 643.972674] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 645.671172] Key type lgssc registered [ 646.734380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.939275] loop: module loaded [ 655.018793] loop0: detected capacity change from 0 to 204800000 [ 655.274763] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.293431] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 655.568433] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.272415] loop0: detected capacity change from 0 to 204800000 [ 662.476940] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.498162] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 662.514306] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 667.865705] loop0: detected capacity change from 0 to 204800000 [ 673.452372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.483768] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.715572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 674.752950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 674.890125] Lustre: lustre-MDT0000: new disk, initializing [ 674.983878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.019609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 679.848252] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 689.231251] loop1: detected capacity change from 0 to 204800000 [ 694.532951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 694.814369] Lustre: lustre-OST07c6: new disk, initializing [ 694.822309] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 694.831641] Lustre: Skipped 1 previous similar message [ 694.841131] Lustre: 26163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 694.921702] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 700.593567] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 704.627663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 704.633689] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 704.695089] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 710.065409] Lustre: 26484:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 710.076238] Lustre: 26484:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 710.084958] Lustre: 26484:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 710.097564] Lustre: 26484:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 710.107320] Lustre: 26484:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 710.113589] Lustre: 26484:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 731.751627] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 731.760754] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 731.768217] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 731.776971] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 731.783040] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 731.789420] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 736.958536] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 736.964113] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 736.972072] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 736.979726] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.986383] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.993679] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.998821] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 737.004654] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.013450] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 737.020641] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 737.025758] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 737.030025] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.236471] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 740.261716] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 740.280943] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 740.295313] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.317810] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 740.329600] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.337151] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 740.347215] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.354070] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 740.358362] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 740.366524] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 740.371807] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.488953] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 745.496826] Lustre: 25146:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 745.504041] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 745.512898] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.519857] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 745.526742] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.533058] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 745.541638] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.547476] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 745.554594] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 745.560860] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 745.565797] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 753.668906] Lustre: 26484:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 753.683347] Lustre: 26484:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 753.690481] Lustre: 26484:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 753.706119] Lustre: 26484:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 753.720857] Lustre: 26484:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 753.732524] Lustre: 26484:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 753.741466] Lustre: 26484:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 753.748210] Lustre: 26484:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 753.758611] Lustre: 26484:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 753.767736] Lustre: 26484:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 753.784972] Lustre: 26484:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 753.796627] Lustre: 26484:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 770.307683] Lustre: 26484:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 770.320299] Lustre: 26484:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 770.332072] Lustre: 26484:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.341792] Lustre: 26484:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 770.355539] Lustre: 26484:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.362596] Lustre: 26484:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 770.371549] Lustre: 26484:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 770.390068] Lustre: 26484:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 770.399168] Lustre: 26484:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.410148] Lustre: 26484:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 770.424600] Lustre: 26484:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.434810] Lustre: 26484:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 806.882039] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 806.897699] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 806.925580] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 812.004340] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 812.629534] Lustre: server umount lustre-OST07c6 complete [ 821.168888] Lustre: server umount lustre-MDT0000 complete [ 836.627608] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 839.264752] Key type lgssc unregistered [ 839.591432] LNet: 28723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.600345] LNetError: 28723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.621600] LNet: Removed LNI 192.168.204.131@tcp [ 840.534514] Key type .llcrypt unregistered [ 840.537209] Key type ._llcrypt unregistered [ 859.450487] Key type ._llcrypt registered [ 859.453616] Key type .llcrypt registered [ 859.558549] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 860.790784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 860.819854] alg: No test for adler32 (adler32-zlib) [ 861.876805] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 862.205115] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 863.911469] Key type lgssc registered [ 865.044872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 873.399437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 873.443970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 874.996610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.734201] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 885.217949] LustreError: 29992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 885.249071] LustreError: 29992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 889.167934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.671030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.533370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 902.241605] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 908.200365] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 914.942315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 915.384505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.520082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 922.329459] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 930.645044] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 951.269871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 951.287110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 951.292276] LustreError: Skipped 1 previous similar message [ 951.313436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 952.526561] Lustre: server umount lustre-OST0000 complete [ 960.482790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.501606] Lustre: Skipped 1 previous similar message [ 960.503789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 960.521361] Lustre: Skipped 1 previous similar message [ 963.004475] Lustre: server umount lustre-MDT0000 complete [ 965.607406] LustreError: 32479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.638385] LustreError: 32479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 966.881872] LustreError: 29973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388444 with bad export cookie 6230764953925979101 [ 966.899469] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.519207] Lustre: server umount lustre-MDT0001 complete [ 988.681974] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 991.648153] Key type lgssc unregistered [ 991.977058] LNet: 33728:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.995549] LNetError: 33728:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.062207] LNet: Removed LNI 192.168.204.131@tcp [ 993.889518] Key type .llcrypt unregistered [ 993.897314] Key type ._llcrypt unregistered [ 1013.726311] Key type ._llcrypt registered [ 1013.728437] Key type .llcrypt registered [ 1013.804466] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1014.573284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1014.657593] alg: No test for adler32 (adler32-zlib) [ 1015.647283] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1015.810980] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1017.479193] Key type lgssc registered [ 1018.943675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.994310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.024544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.585058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.298929] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1038.817076] LustreError: 34999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.856168] LustreError: 34999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1042.375054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.019476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.138515] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1057.266555] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.889506] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1071.553382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.890737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.959696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1079.126708] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1088.000536] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1102.832628] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1102.834557] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1102.845315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1102.858588] Lustre: Skipped 1 previous similar message [ 1106.945508] Lustre: server umount lustre-OST0000 complete [ 1113.057284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.074095] Lustre: Skipped 1 previous similar message [ 1113.079224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1113.084916] Lustre: Skipped 1 previous similar message [ 1117.407522] Lustre: server umount lustre-MDT0000 complete [ 1118.178172] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.188598] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1121.430814] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388599 with bad export cookie 3592468627719822580 [ 1121.440460] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1121.450724] LustreError: 34980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1121.898928] Lustre: server umount lustre-MDT0001 complete [ 1133.182738] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1135.994522] Key type lgssc unregistered [ 1136.264732] LNet: 38556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.271250] LNetError: 38556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.292879] LNet: Removed LNI 192.168.204.131@tcp [ 1137.282530] Key type .llcrypt unregistered [ 1137.284915] Key type ._llcrypt unregistered [ 1162.233191] Key type ._llcrypt registered [ 1162.236311] Key type .llcrypt registered [ 1162.326779] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1163.320057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1163.334706] alg: No test for adler32 (adler32-zlib) [ 1164.404973] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1164.611398] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1166.287683] Key type lgssc registered [ 1167.533208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1175.612554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1175.643949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.183965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1182.978096] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1187.304770] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.324670] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1192.422348] LustreError: 39974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.578410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1193.861468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1198.905179] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1207.162573] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1213.585131] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1221.615294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1222.195256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.244709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1228.557972] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1237.439928] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1244.835722] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 1245.169421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1245.179500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.737471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1248.739919] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1248.752344] Lustre: Skipped 2 previous similar messages [ 1248.783907] Lustre: Skipped 1 previous similar message [ 1249.924736] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 1249.933745] Lustre: Skipped 1 previous similar message [ 1249.960699] LustreError: 42550:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000de238600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1252.047410] Lustre: server umount lustre-MDT0000 complete [ 1253.870752] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1253.892554] LustreError: 39973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1255.690275] LustreError: 39955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388733 with bad export cookie 5548825997052311678 [ 1255.706442] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.710489] LustreError: 39955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1258.980592] LustreError: 39970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1258.981361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.995332] LustreError: 39970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1259.026048] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1262.043858] Lustre: server umount lustre-MDT0001 complete [ 1282.696955] Lustre: server umount lustre-OST0000 complete [ 1298.959571] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1301.658585] Key type lgssc unregistered [ 1302.010798] LNet: 43831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.025144] LNetError: 43831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.043195] LNet: Removed LNI 192.168.204.131@tcp [ 1302.795463] Key type .llcrypt unregistered [ 1302.797369] Key type ._llcrypt unregistered [ 1330.198314] Key type ._llcrypt registered [ 1330.205264] Key type .llcrypt registered [ 1330.371701] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1331.692438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1331.824695] alg: No test for adler32 (adler32-zlib) [ 1332.950146] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1333.200477] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1334.919218] Key type lgssc registered [ 1336.232312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1345.178236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1345.209255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.810921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1351.717645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1356.261095] LustreError: 45250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.272905] LustreError: 45250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1359.879727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.238452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.873570] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1371.895721] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1379.105579] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1388.828838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.308283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.354665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1396.372612] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1404.499911] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1412.065239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1412.071386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.079383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1412.765657] Lustre: lustre-OST0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 1416.160571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.170534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.183959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.283433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.295525] Lustre: Skipped 2 previous similar messages [ 1426.403412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1426.411789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.425349] Lustre: Skipped 2 previous similar messages [ 1426.577582] Lustre: server umount lustre-OST0000 complete [ 1446.883103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.912090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1446.926158] Lustre: Skipped 1 previous similar message [ 1450.943563] Lustre: server umount lustre-MDT0000 complete [ 1452.004035] LustreError: 46260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.020213] LustreError: 46260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1454.603582] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787388932 with bad export cookie 11346219427264670102 [ 1454.617806] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1454.621588] LustreError: 45232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1461.125746] Lustre: server umount lustre-MDT0001 complete [ 1476.089404] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1479.890468] Key type lgssc unregistered [ 1480.218783] LNet: 48960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.229684] LNetError: 48960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.259971] LNet: Removed LNI 192.168.204.131@tcp [ 1481.616565] Key type .llcrypt unregistered [ 1481.620520] Key type ._llcrypt unregistered [ 1513.910731] Key type ._llcrypt registered [ 1513.914282] Key type .llcrypt registered [ 1514.120268] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1515.055736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1515.151812] alg: No test for adler32 (adler32-zlib) [ 1516.177687] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1516.500325] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1518.175193] Key type lgssc registered [ 1519.441203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1529.123625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1529.147848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.069310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.671361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1541.608084] LustreError: 50381:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.641392] LustreError: 50381:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1546.724780] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.246507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.638771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.279808] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1560.492404] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1566.543174] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.795329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.236576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.337522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1581.738538] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1590.274741] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1598.560701] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1602.651897] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1614.961509] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 4h58m10s [ 1616.451589] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1618.405336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.409386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.414273] Lustre: Skipped 1 previous similar message [ 1618.427457] Lustre: Skipped 2 previous similar messages [ 1619.490389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.696088] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 1622.704459] Lustre: Skipped 2 previous similar messages [ 1624.668305] Lustre: server umount lustre-MDT0000 complete [ 1625.574369] LustreError: 50377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.593416] LustreError: 50377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1628.145680] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1629.829565] LustreError: 50375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.444464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.575154] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.857861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.917872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1636.907750] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1640.577370] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1644.338500] Lustre: DEBUG MARKER: Wait for df (36591) ... [ 1645.829969] Lustre: DEBUG MARKER: done [ 1651.172720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.185980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.210276] Lustre: Skipped 1 previous similar message [ 1651.221606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.228838] Lustre: Skipped 1 previous similar message [ 1652.202534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.581597] Lustre: server umount lustre-OST0000 complete [ 1662.436047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.449670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.450684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1662.463364] Lustre: Skipped 3 previous similar messages [ 1667.293874] Lustre: server umount lustre-MDT0000 complete [ 1669.601128] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.626477] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1671.081208] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389149 with bad export cookie 1161564076459592605 [ 1671.098682] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.105897] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1671.601974] Lustre: server umount lustre-MDT0001 complete [ 1684.277555] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1687.352319] Key type lgssc unregistered [ 1687.625914] LNet: 55304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.633507] LNetError: 55304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.649548] LNet: Removed LNI 192.168.204.131@tcp [ 1688.719867] Key type .llcrypt unregistered [ 1688.722753] Key type ._llcrypt unregistered [ 1733.884675] Key type ._llcrypt registered [ 1733.888393] Key type .llcrypt registered [ 1734.052310] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1735.822372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1735.861279] alg: No test for adler32 (adler32-zlib) [ 1737.091439] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1737.410740] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1739.143251] Key type lgssc registered [ 1740.553850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.405079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.460918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.818674] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1750.845360] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1751.040559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.012352] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1765.268755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.357508] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1765.389958] Lustre: 57512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1765.716348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1770.475308] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1777.753751] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1783.507702] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1790.472116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1790.740791] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1790.966925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.211326] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1799.352803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1804.829320] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1813.555861] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 5h01m29s [ 1815.824526] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1823.870250] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1826.845175] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1826.901817] LustreError: 57520:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1846.950240] Lustre: lustre-MDT0000: Client ffbc8c97-955a-4878-883b-5ea90c298bb4 (at 192.168.204.31@tcp) reconnecting [ 1866.927185] LustreError: 57520:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1869.364798] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1872.001937] Lustre: DEBUG MARKER: done [ 1880.909079] Lustre: server umount lustre-OST0000 complete [ 1886.689556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.709779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1886.715282] Lustre: Skipped 1 previous similar message [ 1887.778472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1887.787305] Lustre: Skipped 1 previous similar message [ 1893.858982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.867084] Lustre: Skipped 1 previous similar message [ 1894.956619] Lustre: server umount lustre-MDT0000 complete [ 1897.952177] Lustre: 56838:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389359/real 1787389359] req@0000000013d22c40 x1874213433565440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787389375 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1897.974082] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.986876] Lustre: Skipped 1 previous similar message [ 1899.027296] LustreError: 57506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389377 with bad export cookie 5552872813733276444 [ 1899.038221] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.041281] LustreError: 57506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1899.464058] Lustre: server umount lustre-MDT0001 complete [ 1910.433281] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1913.232790] Key type lgssc unregistered [ 1913.619885] LNet: 61863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.636357] LNetError: 61863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.664958] LNet: Removed LNI 192.168.204.131@tcp [ 1914.525604] Key type .llcrypt unregistered [ 1914.529300] Key type ._llcrypt unregistered [ 1957.988042] Key type ._llcrypt registered [ 1957.990146] Key type .llcrypt registered [ 1958.074392] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1959.062692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.109855] alg: No test for adler32 (adler32-zlib) [ 1960.136467] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 1960.374469] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1962.063528] Key type lgssc registered [ 1963.270123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1966.528108] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1966.539749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.828696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1967.851876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1968.098467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.042585] Lustre: Failing over lustre-MDT0000 [ 1969.291642] Lustre: server umount lustre-MDT0000 complete [ 1976.040930] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 05:04:12 (1787389452) [ 1982.741780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.218578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.641428] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1996.271594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.532858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1996.565888] Lustre: 64662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1996.993889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2002.738504] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2013.127841] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2021.218603] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2027.983884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.300797] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2028.598860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.911150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2036.061713] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2043.799584] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2049.301813] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2055.651800] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.655798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.663474] Lustre: Skipped 2 previous similar messages [ 2055.669400] Lustre: Skipped 1 previous similar message [ 2060.771615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2060.782316] Lustre: Skipped 3 previous similar messages [ 2061.959166] Lustre: server umount lustre-MDT0000 complete [ 2065.471598] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389543 with bad export cookie 18161960950653147536 [ 2065.482844] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.485790] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2065.852556] Lustre: server umount lustre-MDT0001 complete [ 2069.789365] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2074.605142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2081.250300] Lustre: 63586:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389543/real 1787389543] req@000000009906e337 x1874213667382144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.300781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.078686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.391374] Lustre: 63584:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389549/real 1787389549] req@00000000a3470fc4 x1874213667382272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.527706] LustreError: 63583:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051c1cbf0 x1874213667383040/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.166372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.277084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2092.199179] Lustre: 63584:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389553/real 1787389553] req@000000008e857a3c x1874213667382528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389569 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.117344] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2097.124863] Lustre: 63587:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389558/real 1787389558] req@0000000095011ccf x1874213667382656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2097.643966] LustreError: 68431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2097.659853] LustreError: 68431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2102.759456] LustreError: 68433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2102.790881] LustreError: 68433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2104.748628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2105.122259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2110.255741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2110.383500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2118.279961] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2124.159985] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2130.917342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2130.921140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.943662] Lustre: Skipped 2 previous similar messages [ 2130.949522] LustreError: 68431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.960329] LustreError: 68431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2131.100814] Lustre: server umount lustre-MDT0000 complete [ 2134.908065] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389612 with bad export cookie 18161960950653149811 [ 2134.924684] LustreError: 64653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2134.926362] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.326045] Lustre: server umount lustre-MDT0001 complete [ 2141.926254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2151.559335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.415172] Lustre: 63584:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389614/real 1787389614] req@000000006b6f798a x1874213667408384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2152.448928] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.992206] LustreError: 66511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.013092] LustreError: 66511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2161.121772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.213901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2162.144904] Lustre: 63585:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787389623/real 1787389623] req@000000009d952049 x1874213667408896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787389639 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2162.193449] Lustre: 63585:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2165.779153] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2167.271795] LustreError: 71113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.286426] LustreError: 71113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2171.379309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2171.384755] Lustre: Skipped 1 previous similar message [ 2173.998674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.813760] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2179.571568] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2186.443895] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2192.520315] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2200.035232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2200.038613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.040048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.064147] Lustre: Skipped 2 previous similar messages [ 2205.157428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2205.169975] Lustre: Skipped 4 previous similar messages [ 2205.991069] Lustre: server umount lustre-MDT0000 complete [ 2209.389947] LustreError: 65396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389687 with bad export cookie 18161960950653151190 [ 2209.390916] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.423938] LustreError: 65396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2210.050763] Lustre: server umount lustre-MDT0001 complete [ 2217.441765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2220.947927] Lustre: DEBUG MARKER: files compared the same [ 2232.345793] Lustre: server umount lustre-OST0000 complete [ 2248.175064] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2250.999547] Key type lgssc unregistered [ 2251.237633] LNet: 74388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.251191] LNetError: 74388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.275705] LNet: Removed LNI 192.168.204.131@tcp [ 2252.011742] Key type .llcrypt unregistered [ 2252.013953] Key type ._llcrypt unregistered [ 2276.253744] Key type ._llcrypt registered [ 2276.256451] Key type .llcrypt registered [ 2276.326054] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2277.379872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2277.449645] alg: No test for adler32 (adler32-zlib) [ 2278.606385] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2278.865576] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2280.519178] Key type lgssc registered [ 2281.922534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2290.278896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2290.316044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.877492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.273234] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug +malloc all [ 2301.421775] LustreError: 75805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.465678] LustreError: 75805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2306.530749] LustreError: 75804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2307.779924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.125098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2312.684173] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug +malloc all [ 2320.660229] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2326.392947] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2332.975186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.379943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.454277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2340.402705] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug +malloc all [ 2348.440443] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2360.293187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.302820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.319377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.338332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2362.338864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2362.343194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.369121] Lustre: Skipped 1 previous similar message [ 2364.303490] Lustre: server umount lustre-OST0000 complete [ 2369.505535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2369.516834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.539896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.578131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.586572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2374.862066] Lustre: server umount lustre-MDT0000 complete [ 2377.696058] LustreError: 75804:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.712714] LustreError: 75804:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2378.675057] LustreError: 75785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787389856 with bad export cookie 17122720282034245154 [ 2378.690687] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.091393] Lustre: server umount lustre-MDT0001 complete [ 2389.255507] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2391.803599] Key type lgssc unregistered [ 2392.017652] LNet: 79339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.029418] LNetError: 79339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2393.067232] LNet: Removed LNI 192.168.204.131@tcp [ 2393.909436] Key type .llcrypt unregistered [ 2393.911686] Key type ._llcrypt unregistered [ 2417.834405] Key type ._llcrypt registered [ 2417.838869] Key type .llcrypt registered [ 2417.974504] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2418.866584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2418.881885] alg: No test for adler32 (adler32-zlib) [ 2420.013739] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2420.306943] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2421.975678] Key type lgssc registered [ 2423.009520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2430.534788] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.549569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.111742] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2451.231744] LustreError: 80729:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 2478.047232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.035609] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2491.332244] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2500.772518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.927100] Lustre: *** cfs_fail_loc=706, val=0*** [ 2501.280152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.369158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2505.782777] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2506.724281] LustreError: 80754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.757237] LustreError: 80754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2511.847803] LustreError: 80755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.878417] LustreError: 80754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.914533] LustreError: 80754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2514.810994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.261580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2519.744125] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2526.699135] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2531.505023] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2535.908195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2535.913189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.922295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2535.944529] Lustre: Skipped 1 previous similar message [ 2539.571466] Lustre: server umount lustre-OST0000 complete [ 2546.147623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2546.148898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.169815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.182285] Lustre: Skipped 1 previous similar message [ 2546.188191] Lustre: Skipped 1 previous similar message [ 2549.704272] Lustre: server umount lustre-MDT0000 complete [ 2551.266208] LustreError: 81815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2551.288394] LustreError: 81815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2554.451734] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390032 with bad export cookie 370612218027847353 [ 2554.459825] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2554.463364] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2555.019170] Lustre: server umount lustre-MDT0001 complete [ 2568.616109] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2572.458975] Key type lgssc unregistered [ 2572.927569] LNet: 84344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2572.934369] LNetError: 84344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.992583] LNet: Removed LNI 192.168.204.131@tcp [ 2574.911552] Key type .llcrypt unregistered [ 2574.916527] Key type ._llcrypt unregistered [ 2601.171368] Key type ._llcrypt registered [ 2601.175612] Key type .llcrypt registered [ 2601.313826] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2602.412229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2602.489702] alg: No test for adler32 (adler32-zlib) [ 2603.543115] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2603.775702] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2605.471199] Key type lgssc registered [ 2606.736508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2615.305336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2615.334512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.203781] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2629.222363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.665798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2634.273617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2634.741947] LustreError: 86451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2634.777181] LustreError: 86451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2639.844054] LustreError: 86449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.869615] LustreError: 86449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2644.963423] LustreError: 86451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.978517] LustreError: 86451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2645.362640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2645.879129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.112278] LustreError: 86450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.148184] LustreError: 86450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2653.780978] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2659.324489] LustreError: 86454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.356338] LustreError: 86454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2662.937040] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.946794] Lustre: MGS: Client f1edca5c-aa42-401d-8b38-96b473a6281d (at 0@lo) reconnecting [ 2662.962563] LustreError: 86465:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2663.244354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.325114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2667.638474] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2683.885810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.886972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2683.928273] LustreError: Skipped 1 previous similar message [ 2683.929858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2683.936688] Lustre: Skipped 1 previous similar message [ 2689.000130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.009057] Lustre: Skipped 1 previous similar message [ 2689.332320] Lustre: server umount lustre-OST0000 complete [ 2694.112636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2694.118658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.137180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.150596] Lustre: Skipped 2 previous similar messages [ 2694.162662] Lustre: Skipped 1 previous similar message [ 2699.234737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.246543] Lustre: Skipped 1 previous similar message [ 2699.538185] Lustre: server umount lustre-MDT0000 complete [ 2703.278073] Lustre: server umount lustre-MDT0001 complete [ 2707.408919] Lustre: server umount MGS complete [ 2719.363348] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 05:16:35 (1787390195) [ 2749.024965] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 2758.151169] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2797.647505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.023461] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2813.354177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.502982] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2813.577976] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2813.645771] Lustre: lustre-MDT0001: new disk, initializing [ 2813.720188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2813.744876] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2813.749187] Lustre: 93644:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2813.754025] LustreError: 93644:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2819.048267] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2831.199068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2831.566357] Lustre: lustre-OST0000: new disk, initializing [ 2831.571611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2831.579568] Lustre: 94410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2831.681907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.314267] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2839.648945] Lustre: 94832:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a3ad733f x1874214341616384/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787390333 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2845.885889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2853.099609] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.120346] Lustre: MGS: Client a9822bf2-b25a-44a3-8196-001f0841d15d (at 0@lo) reconnecting [ 2853.193692] Lustre: 92911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2853.231894] LustreError: 92911:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2853.245476] LustreError: 92911:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2853.319945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2853.420420] Lustre: lustre-MDT0000: new disk, initializing [ 2853.575078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.606799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2853.805067] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2854.698256] Lustre: 95478:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000007c785fe6 x1874214341622912/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787390348 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2859.527760] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2860.006082] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.792276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2863.811750] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2864.085213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2886.121651] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2886.123692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2886.131462] Lustre: Skipped 1 previous similar message [ 2886.137689] Lustre: Skipped 1 previous similar message [ 2891.990899] Lustre: server umount lustre-OST0000 complete [ 2896.353334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.359527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.373411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2896.397477] Lustre: Skipped 1 previous similar message [ 2896.409645] Lustre: Skipped 3 previous similar messages [ 2902.041731] Lustre: server umount lustre-MDT0000 complete [ 2912.535567] Lustre: server umount lustre-MDT0001 complete [ 2916.251408] Lustre: server umount MGS complete [ 2929.463928] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 05:20:06 (1787390406) [ 2948.797384] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2951.639813] Key type lgssc unregistered [ 2951.908197] LNet: 97977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.915254] LNetError: 97977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.937979] LNet: Removed LNI 192.168.204.131@tcp [ 2953.922290] Key type .llcrypt unregistered [ 2953.925992] Key type ._llcrypt unregistered [ 2970.261424] Key type ._llcrypt registered [ 2970.263905] Key type .llcrypt registered [ 2970.365393] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2971.184519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2971.252957] alg: No test for adler32 (adler32-zlib) [ 2972.430787] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 2972.718907] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2974.391576] Key type lgssc registered [ 2975.836150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2984.203967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2984.221180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2985.551747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2989.973457] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2995.682129] LustreError: 99220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2995.693459] LustreError: 99220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2998.690592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.044746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3002.985510] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3014.669588] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3021.292418] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3028.726215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.051244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3030.086353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3036.151745] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3044.035727] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3054.563284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.567360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.570231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3054.580843] Lustre: Skipped 1 previous similar message [ 3058.490881] Lustre: server umount lustre-OST0000 complete [ 3064.800723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.804424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.817187] Lustre: Skipped 1 previous similar message [ 3064.824160] Lustre: Skipped 1 previous similar message [ 3068.477317] Lustre: server umount lustre-MDT0000 complete [ 3069.932037] LustreError: 99221:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.946817] LustreError: 99221:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3072.863575] LustreError: 99200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390550 with bad export cookie 7549052182588545822 [ 3072.868286] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.873723] LustreError: 99200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3073.405791] Lustre: server umount lustre-MDT0001 complete [ 3084.866478] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3087.713692] Key type lgssc unregistered [ 3087.997899] LNet: 102755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3088.012435] LNetError: 102755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3088.038265] LNet: Removed LNI 192.168.204.131@tcp [ 3089.003318] Key type .llcrypt unregistered [ 3089.006943] Key type ._llcrypt unregistered [ 3104.874730] Key type ._llcrypt registered [ 3104.877690] Key type .llcrypt registered [ 3104.997648] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3107.054544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3107.089549] alg: No test for adler32 (adler32-zlib) [ 3108.405422] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 3108.826001] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3110.471891] Key type lgssc registered [ 3112.035609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3120.364285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3120.378135] LustreError: 103838:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3120.387877] LustreError: 103839:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3120.409797] LustreError: 103838:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 3120.417804] LustreError: 103838:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3120.425070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3120.431155] LustreError: 103838:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3122.032682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.897144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3130.358143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3135.343103] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3142.200495] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing lsmod [ 3145.855704] LustreError: 105043:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3145.879878] LustreError: 105044:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3145.887557] LustreError: 105043:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3145.895646] LustreError: 105043:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3145.911429] LustreError: 105043:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3145.923907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3146.467449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3147.511637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3150.816641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.827987] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.843975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.866250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3152.867181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.890591] Lustre: Skipped 1 previous similar message [ 3152.911293] Lustre: Skipped 2 previous similar messages [ 3154.982700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3154.992246] Lustre: Skipped 2 previous similar messages [ 3156.282825] Lustre: server umount lustre-MDT0000 complete [ 3160.034460] LustreError: 103875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.067294] LustreError: 103875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3160.141052] LustreError: 103855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390638 with bad export cookie 12956559594985791863 [ 3160.156236] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3161.183387] LustreError: 104306:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c602c083 x1874214871128448/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3161.389568] Lustre: server umount lustre-MDT0001 complete [ 3175.233037] Lustre: server umount lustre-OST0000 complete [ 3185.812479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.474051] LustreError: 106109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.564187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.779371] LustreError: 106110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.121687] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3196.898902] LustreError: 106109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.020574] LustreError: 106110:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3202.284772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.619159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3208.163945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3216.861494] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3223.317444] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3231.483786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.965631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.256661] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3247.088449] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3256.567734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3267.048517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.061158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.067317] Lustre: Skipped 1 previous similar message [ 3271.592908] Lustre: server umount lustre-OST0000 complete [ 3277.292983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.297266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3277.312149] Lustre: Skipped 1 previous similar message [ 3277.324473] Lustre: Skipped 2 previous similar messages [ 3282.736559] Lustre: server umount lustre-MDT0000 complete [ 3287.523672] LustreError: 107055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3287.556180] LustreError: 107055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3288.119836] LustreError: 106090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390766 with bad export cookie 12956559594985792794 [ 3288.129694] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.130285] LustreError: 106090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3288.677921] Lustre: server umount lustre-MDT0001 complete [ 3301.916628] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3305.590354] Key type lgssc unregistered [ 3305.998444] LNet: 109669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3306.009374] LNetError: 109669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3306.031645] LNet: Removed LNI 192.168.204.131@tcp [ 3307.032797] Key type .llcrypt unregistered [ 3307.038188] Key type ._llcrypt unregistered [ 3331.454563] Key type ._llcrypt registered [ 3331.457934] Key type .llcrypt registered [ 3331.579291] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3332.749974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3332.901533] alg: No test for adler32 (adler32-zlib) [ 3334.062255] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 3334.340614] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3336.039207] Key type lgssc registered [ 3337.115041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3345.391724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3345.406232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.814386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.096346] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3356.136424] LustreError: 111087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3356.162636] LustreError: 111087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3361.220090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.250086] LustreError: 111088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3361.484131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.674061] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3374.552963] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3381.195454] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3389.659694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3390.132305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.192576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3397.128643] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3406.300924] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3413.577875] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3418.641349] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3427.296310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3427.310268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.311514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3427.333200] Lustre: Skipped 1 previous similar message [ 3429.763242] Lustre: server umount lustre-OST0000 complete [ 3433.449498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3433.458814] LustreError: Skipped 1 previous similar message [ 3433.464807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.486678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3433.496147] Lustre: Skipped 1 previous similar message [ 3437.539162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3437.539461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3439.532906] Lustre: server umount lustre-MDT0000 complete [ 3442.662281] LustreError: 111083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.688636] LustreError: 111083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3443.643058] LustreError: 111068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787390921 with bad export cookie 612128615980153598 [ 3443.653388] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3443.657039] LustreError: 111068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3444.007119] Lustre: server umount lustre-MDT0001 complete [ 3456.098728] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3459.011448] Key type lgssc unregistered [ 3459.347602] LNet: 114726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3459.362961] LNetError: 114726:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3459.388056] LNet: Removed LNI 192.168.204.131@tcp [ 3460.619972] Key type .llcrypt unregistered [ 3460.622500] Key type ._llcrypt unregistered [ 3477.772377] Key type ._llcrypt registered [ 3477.775388] Key type .llcrypt registered [ 3477.895694] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3478.942346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3478.970205] alg: No test for adler32 (adler32-zlib) [ 3480.103662] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 3480.388572] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3482.071533] Key type lgssc registered [ 3483.357229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3491.539827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3491.567955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.054592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3498.259174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3503.586418] LustreError: 115953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.602470] LustreError: 115953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3508.707384] LustreError: 115954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3509.175093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.754893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.588298] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3526.566196] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3533.607628] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3540.975038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3541.516478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3548.245315] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3556.754475] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3564.105660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3575.265093] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3575.275200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3575.282545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3579.364176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.367736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3579.389683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3579.405675] Lustre: Skipped 1 previous similar message [ 3581.507249] Lustre: server umount lustre-OST0000 complete [ 3586.546337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3586.555655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.578640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.601168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3589.601673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.958881] Lustre: server umount lustre-MDT0000 complete [ 3594.725198] LustreError: 116894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3594.741410] LustreError: 116894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3596.367436] LustreError: 115934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391074 with bad export cookie 12508814648588888273 [ 3596.368260] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3596.377645] LustreError: 115934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3596.767694] Lustre: server umount lustre-MDT0001 complete [ 3608.737853] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3611.550209] Key type lgssc unregistered [ 3611.944824] LNet: 119513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3611.959773] LNetError: 119513:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3611.989418] LNet: Removed LNI 192.168.204.131@tcp [ 3613.033309] Key type .llcrypt unregistered [ 3613.043301] Key type ._llcrypt unregistered [ 3638.469978] Key type ._llcrypt registered [ 3638.472803] Key type .llcrypt registered [ 3638.618072] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3640.235956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3640.344680] alg: No test for adler32 (adler32-zlib) [ 3641.565871] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 3641.911912] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3643.679172] Key type lgssc registered [ 3645.158786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3653.538885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3653.565709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3655.097341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.395987] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3665.380649] LustreError: 120930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.401186] LustreError: 120930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3667.381148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.769583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3672.630873] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3680.710104] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3686.438479] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3693.403568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3693.758892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3699.931310] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3700.925383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3707.637874] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3718.778557] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3723.331728] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3723.337445] Lustre: Skipped 2 previous similar messages [ 3726.380130] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3726.427213] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3727.806554] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3727.815207] Lustre: Skipped 2 previous similar messages [ 3734.590570] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3734.594126] Lustre: Skipped 1 previous similar message [ 3739.391726] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.31@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3739.398905] Lustre: Skipped 2 previous similar messages [ 3743.271263] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.31@tcp 192.168.0.[2,10]@tcp [ 3751.127586] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.31/24@tcp in log lustre-MDT0000 [ 3751.147933] Lustre: Skipped 2 previous similar messages [ 3753.240780] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.31/24@tcp [ 3753.244751] Lustre: Skipped 1 previous similar message [ 3764.737124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3764.739840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.742478] LustreError: Skipped 2 previous similar messages [ 3764.764431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3766.277956] Lustre: server umount lustre-OST0000 complete [ 3770.343717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3770.349370] LustreError: Skipped 3 previous similar messages [ 3770.355621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3770.369757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.921901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.924055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.041757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.052842] Lustre: Skipped 2 previous similar messages [ 3784.160952] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3784.166909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3784.178563] Lustre: Skipped 1 previous similar message [ 3784.384943] Lustre: server umount lustre-MDT0000 complete [ 3784.735573] Lustre: 120241:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787391246/real 1787391246] req@000000006316c673 x1874215430065536/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787391262 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3784.761306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.697325] LustreError: 120911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391266 with bad export cookie 7252333117775096339 [ 3788.699949] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.703567] LustreError: 120911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3789.098360] Lustre: server umount lustre-MDT0001 complete [ 3799.062291] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3801.931889] Key type lgssc unregistered [ 3802.229172] LNet: 125785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3802.242237] LNetError: 125785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3802.275567] LNet: Removed LNI 192.168.204.131@tcp [ 3803.122678] Key type .llcrypt unregistered [ 3803.125090] Key type ._llcrypt unregistered [ 3830.899371] Key type ._llcrypt registered [ 3830.903320] Key type .llcrypt registered [ 3831.109392] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3832.841241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3832.936320] alg: No test for adler32 (adler32-zlib) [ 3834.027587] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 3834.294822] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 3835.999193] Key type lgssc registered [ 3837.371111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3845.910867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3845.929556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3847.571103] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3847.600978] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.31/24@tcp [ 3847.611455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.399717] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3857.891176] LustreError: 127350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3857.914203] LustreError: 127350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3861.314881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.628288] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3861.647283] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.31/24@tcp [ 3861.708498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.616943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3874.151869] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3880.542730] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3888.282214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.743187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.865163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3895.583833] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3903.113446] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3918.827728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3918.843889] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.854770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3921.892400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3921.902458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3921.905439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.349207] Lustre: server umount lustre-OST0000 complete [ 3928.544525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.557758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.576523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.588939] Lustre: Skipped 1 previous similar message [ 3932.134852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.136672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.258187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3937.267871] Lustre: Skipped 1 previous similar message [ 3942.367615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3942.663648] Lustre: server umount lustre-MDT0000 complete [ 3943.395090] LustreError: 127350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3943.419660] LustreError: 127350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3947.384299] LustreError: 127330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391425 with bad export cookie 2976140769129915403 [ 3947.399583] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3947.790715] Lustre: server umount lustre-MDT0001 complete [ 3959.794621] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 3963.090533] Key type lgssc unregistered [ 3963.497594] LNet: 130942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3963.524647] LNetError: 130942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3963.549062] LNet: Removed LNI 192.168.204.131@tcp [ 3964.772264] Key type .llcrypt unregistered [ 3964.777259] Key type ._llcrypt unregistered [ 3999.790525] Key type ._llcrypt registered [ 3999.808019] Key type .llcrypt registered [ 3999.980244] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4020.253755] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4021.978516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4022.008366] alg: No test for adler32 (adler32-zlib) [ 4022.974634] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 4023.370522] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4025.191227] Key type lgssc registered [ 4026.567239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4073.228246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4073.264439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4074.621903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4074.686069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4074.837954] Lustre: lustre-MDT0000: new disk, initializing [ 4074.892170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4074.906473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4080.643199] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4093.220934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.349833] Lustre: 134350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4093.408478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4093.413000] Lustre: Skipped 1 previous similar message [ 4093.493798] Lustre: lustre-MDT0001: new disk, initializing [ 4093.658439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4093.729228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4093.750782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4099.024936] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4109.307364] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4114.541929] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4122.041723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4122.327395] Lustre: lustre-OST0000: new disk, initializing [ 4122.333689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4122.344214] Lustre: 136317:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4122.441839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.142124] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4131.921881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4131.930400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4132.043958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4138.939366] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4143.360879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4143.652545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4148.505316] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4148.713857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4152.817921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4152.834114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.863201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4152.865967] Lustre: Skipped 1 previous similar message [ 4152.886528] Lustre: Skipped 1 previous similar message [ 4156.660939] Lustre: server umount lustre-OST0000 complete [ 4160.481996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4160.490410] LustreError: Skipped 1 previous similar message [ 4160.495888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.510950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.045142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4163.047706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.239662] Lustre: server umount lustre-MDT0000 complete [ 4171.786943] LustreError: 134341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391649 with bad export cookie 2870552775810680667 [ 4171.793067] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.806713] LustreError: 134341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4172.342900] Lustre: server umount lustre-MDT0001 complete [ 4182.447454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.851117] LustreError: 138535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.876029] LustreError: 138535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4182.936738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4187.992904] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4188.136618] LustreError: 138536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.231979] LustreError: 138535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4197.349722] LustreError: 138536:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4198.512755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4198.809878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4204.343942] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4213.558795] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4220.209240] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4228.103868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4228.524525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.521802] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4243.349622] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4248.943809] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4252.442843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4257.756365] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4258.069400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4267.163899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4267.334380] Lustre: lustre-OST0001: new disk, initializing [ 4267.337432] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4267.342551] Lustre: 141931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4267.434665] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4273.135869] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4274.296801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4274.313138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4274.459156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4281.897918] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4282.185051] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4288.064309] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4288.402822] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4294.944067] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894c02c2d800.ost_server_uuid 50 [ 4296.892461] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff894c02c2d800.ost_server_uuid in FULL state after 0 sec [ 4308.874378] Lustre: server umount lustre-OST0001 complete [ 4310.498272] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4310.506828] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4310.510424] LustreError: Skipped 1 previous similar message [ 4310.525573] LustreError: 140361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4310.531047] Lustre: Skipped 1 previous similar message [ 4310.555602] LustreError: 140361:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4313.579819] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4313.589521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4313.597267] Lustre: Skipped 1 previous similar message [ 4315.616910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4315.636516] Lustre: Skipped 1 previous similar message [ 4319.429958] Lustre: server umount lustre-OST0000 complete [ 4325.871541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4325.887974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4325.897094] Lustre: Skipped 1 previous similar message [ 4325.903801] Lustre: Skipped 2 previous similar messages [ 4330.001670] Lustre: server umount lustre-MDT0000 complete [ 4330.988146] LustreError: 138530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.016673] LustreError: 138530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4334.486146] LustreError: 138518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787391812 with bad export cookie 2870552775810681542 [ 4334.488305] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.500115] LustreError: 138518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4334.912590] Lustre: server umount lustre-MDT0001 complete [ 4358.168435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4360.945897] Key type lgssc unregistered [ 4361.282025] LNet: 144933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4361.289828] LNetError: 144933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4361.302704] LNet: Removed LNI 192.168.204.131@tcp [ 4362.184234] Key type .llcrypt unregistered [ 4362.186840] Key type ._llcrypt unregistered [ 4408.552931] Key type ._llcrypt registered [ 4408.555154] Key type .llcrypt registered [ 4408.655839] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4423.993733] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4425.040835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4425.056211] alg: No test for adler32 (adler32-zlib) [ 4426.209586] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 4426.656503] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4428.439539] Key type lgssc registered [ 4430.009478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4474.692261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4474.736936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4475.957971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4475.987948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4476.087192] Lustre: lustre-MDT0000: new disk, initializing [ 4476.190768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.225961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4481.212510] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4494.598985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4494.804981] Lustre: 149100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4494.926996] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4494.934150] Lustre: Skipped 1 previous similar message [ 4495.078641] Lustre: lustre-MDT0001: new disk, initializing [ 4495.257333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4495.312132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4495.321939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4500.260799] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4511.892704] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.645621] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4525.169226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4525.503896] Lustre: lustre-OST0000: new disk, initializing [ 4525.507499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4525.513673] Lustre: 151070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4525.615127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4531.142716] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4534.856082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4534.870830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4534.989699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4540.692983] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4546.697772] Lustre: Failing over lustre-OST0000 [ 4546.883831] Lustre: server umount lustre-OST0000 complete [ 4563.935565] Lustre: 147324:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392026/real 1787392026] req@00000000ed14d0c2 x1874216253162496/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787392042 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4563.959529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.779321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4566.047111] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392028/real 1787392028] req@000000003b3434e1 x1874216253163264/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787392044 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4566.047318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.073649] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4566.159889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4566.183042] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4568.173684] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4571.103095] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392033/real 1787392033] req@000000007065e5d4 x1874216253164800/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787392049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4571.117729] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4571.593896] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4574.427749] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4574.429381] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4575.872540] Lustre: Failing over lustre-MDT0000 [ 4576.209919] Lustre: server umount lustre-MDT0000 complete [ 4576.935727] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787392038/real 1787392038] req@00000000e1b4336a x1874216253165824/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787392054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4576.961422] Lustre: 147326:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4577.248792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4577.265496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.820618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.854180] Lustre: Skipped 1 previous similar message [ 4584.577890] LustreError: 149109:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.623535] LustreError: 149109:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4589.695129] LustreError: 151080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4589.703076] LustreError: 151080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4594.818786] LustreError: 151080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.849459] LustreError: 151080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4594.867706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4594.986133] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4595.274065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.322987] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4599.795443] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4599.939423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4600.300429] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4600.311643] Lustre: Skipped 1 previous similar message [ 4600.346365] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4600.390682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4610.532031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4610.532480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4610.535436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4610.560874] Lustre: Skipped 1 previous similar message [ 4615.653555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4615.660068] Lustre: Skipped 2 previous similar messages [ 4620.777553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4620.783664] Lustre: Skipped 1 previous similar message [ 4621.279193] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4621.384419] Lustre: server umount lustre-OST0000 complete [ 4625.895724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4625.904689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4625.905125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4625.915476] Lustre: Skipped 1 previous similar message [ 4625.952452] Lustre: Skipped 1 previous similar message [ 4631.011448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4631.018576] Lustre: Skipped 1 previous similar message [ 4631.590973] Lustre: server umount lustre-MDT0000 complete [ 4635.703142] LustreError: 149092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392113 with bad export cookie 2988479206148915867 [ 4635.704186] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.718475] LustreError: 149092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.161491] Lustre: server umount lustre-MDT0001 complete [ 4647.053694] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 4649.757980] Key type lgssc unregistered [ 4650.087812] LNet: 154729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4650.096275] LNetError: 154729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4650.110807] LNet: Removed LNI 192.168.204.131@tcp [ 4650.922709] Key type .llcrypt unregistered [ 4650.925375] Key type ._llcrypt unregistered [ 4688.704127] Key type ._llcrypt registered [ 4688.705698] Key type .llcrypt registered [ 4688.842308] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4690.003767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4690.064435] alg: No test for adler32 (adler32-zlib) [ 4691.252939] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 4691.453872] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 4693.127600] Key type lgssc registered [ 4694.542431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4703.915795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4703.928559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.344432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4710.364947] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4715.490104] LustreError: 156249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4715.524917] LustreError: 156249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4719.766889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4720.608785] LustreError: 156250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4723.674268] LustreError: 156278:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4723.683257] LustreError: 156278:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4723.691538] LustreError: 156278:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4724.023244] Lustre: server umount lustre-MDT0000 complete [ 4740.855875] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4750.049284] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4787.329236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.565148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4787.586625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4787.646958] Lustre: lustre-MDT0000: new disk, initializing [ 4787.718241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.743707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4792.204669] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4802.775564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.921909] Lustre: 159584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4803.007859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4803.017189] Lustre: Skipped 1 previous similar message [ 4803.085280] Lustre: lustre-MDT0001: new disk, initializing [ 4803.162283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4803.197422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4803.209152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4807.901921] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4817.404414] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4822.845910] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.123179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4830.421250] Lustre: lustre-OST0000: new disk, initializing [ 4830.424832] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4830.432474] Lustre: 161555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4830.559529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4836.798927] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4838.969103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4838.982047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4839.087161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4845.841889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4849.547300] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4851.282912] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 05:52:08 (1787392328) [ 5125.603354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5125.617654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5125.630789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.121586] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.133244] Lustre: Skipped 1 previous similar message [ 5126.140349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.149452] Lustre: Skipped 2 previous similar messages [ 5130.819855] Lustre: server umount lustre-MDT0000 complete [ 5131.239355] LustreError: 159590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.260754] LustreError: 159590:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5134.429181] LustreError: 160381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392612 with bad export cookie 4270689719373677732 [ 5134.430249] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5134.451364] LustreError: 160381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5136.352805] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.365415] LustreError: 160655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.385639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5139.360612] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5140.875890] Lustre: server umount lustre-MDT0001 complete [ 5159.391156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5159.582140] Lustre: server umount lustre-OST0000 complete [ 5168.110691] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 5177.285171] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5218.346114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.625197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5218.679541] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5218.796948] Lustre: lustre-MDT0000: new disk, initializing [ 5218.914586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.949967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.003094] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5237.081832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.292593] Lustre: 166218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5237.348933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5237.352870] Lustre: Skipped 1 previous similar message [ 5237.417895] Lustre: lustre-MDT0001: new disk, initializing [ 5237.485464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5237.511094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5237.516386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5242.010252] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5251.425193] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.915932] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5261.540233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5261.788697] Lustre: lustre-OST0000: new disk, initializing [ 5261.793937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5261.805434] Lustre: 168185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5261.875590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.497706] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5269.555126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5269.566683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5269.663450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5274.943290] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5278.975557] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5279.156416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5283.526670] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5283.727823] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5290.464711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5290.470531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.480645] LustreError: Skipped 1 previous similar message [ 5290.481599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.493882] Lustre: Skipped 1 previous similar message [ 5300.192691] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5300.546203] Lustre: server umount lustre-OST0000 complete [ 5304.289688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5304.297157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5304.316178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5304.322591] Lustre: Skipped 3 previous similar messages [ 5310.545481] Lustre: server umount lustre-MDT0000 complete [ 5310.945421] LustreError: 167304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5310.964055] LustreError: 167304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5314.030328] LustreError: 166211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392792 with bad export cookie 4270689719373836562 [ 5314.033618] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5314.042029] LustreError: 166211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5314.394502] Lustre: server umount lustre-MDT0001 complete [ 5322.757976] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 05:59:59 (1787392799) [ 5338.351240] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 5347.238391] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5384.565457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.706540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5384.728249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5384.782499] Lustre: lustre-MDT0000: new disk, initializing [ 5384.830803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5384.846990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5389.090795] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5399.322978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5399.511544] Lustre: 172951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5399.799687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5404.992397] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5414.488395] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5420.734041] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5427.145261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5427.385113] Lustre: lustre-OST0000: new disk, initializing [ 5427.387833] Lustre: Skipped 1 previous similar message [ 5427.391879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5427.396300] Lustre: Skipped 2 previous similar messages [ 5427.401487] Lustre: 174930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5433.328595] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5436.974242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5436.977860] Lustre: Skipped 1 previous similar message [ 5436.989407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5437.122755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.949791] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5452.769116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.779627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.782778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5452.782786] Lustre: Skipped 2 previous similar messages [ 5452.794542] Lustre: Skipped 2 previous similar messages [ 5464.034411] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5464.235284] Lustre: server umount lustre-OST0000 complete [ 5474.305698] Lustre: server umount lustre-MDT0000 complete [ 5478.369126] LustreError: 172943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392956 with bad export cookie 4270689719373837444 [ 5478.386748] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.731638] Lustre: server umount lustre-MDT0001 complete [ 5489.903589] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 06:02:46 (1787392966) [ 5506.340487] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 5518.265886] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5556.466559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.808511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5556.866697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5556.994885] Lustre: lustre-MDT0000: new disk, initializing [ 5557.100417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.109801] Lustre: Skipped 2 previous similar messages [ 5557.157686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5562.041945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5572.351846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.524909] Lustre: 179256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5572.599626] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5572.605407] Lustre: Skipped 9 previous similar messages [ 5572.810086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5577.422153] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5588.550590] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5595.224197] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5602.551444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5602.787800] Lustre: 181235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5609.383904] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5610.610431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5610.777243] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5618.622511] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5626.341637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5626.344272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.353246] LustreError: Skipped 1 previous similar message [ 5626.371588] Lustre: Skipped 3 previous similar messages [ 5639.135254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5639.298659] Lustre: server umount lustre-OST0000 complete [ 5644.769928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5644.787513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5644.797199] Lustre: Skipped 6 previous similar messages [ 5649.727729] Lustre: server umount lustre-MDT0000 complete [ 5651.944362] LustreError: 179263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5651.957024] LustreError: 179263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5654.375309] LustreError: 179249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393132 with bad export cookie 4270689719373838529 [ 5654.397942] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5654.410553] LustreError: 179249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5654.932168] Lustre: server umount lustre-MDT0001 complete [ 5666.286922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 5669.102925] Key type lgssc unregistered [ 5669.424959] LNet: 183094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5669.431937] LNetError: 183094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5669.446573] LNet: Removed LNI 192.168.204.131@tcp [ 5670.184102] Key type .llcrypt unregistered [ 5670.188479] Key type ._llcrypt unregistered [ 5693.358958] Key type ._llcrypt registered [ 5693.362904] Key type .llcrypt registered [ 5693.485490] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5694.351457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5694.462573] alg: No test for adler32 (adler32-zlib) [ 5695.471731] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 5695.672512] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 5697.319136] Key type lgssc registered [ 5698.375238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5706.232266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5706.275681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.842204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.604781] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5716.972277] LustreError: 184509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.994748] LustreError: 184509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5721.769451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.165257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5725.952781] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5732.973263] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5738.732879] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5745.673921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.988311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5751.983587] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5752.197547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5760.065432] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5783.008644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5783.018029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.037625] LustreError: Skipped 1 previous similar message [ 5783.038844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.068388] Lustre: Skipped 1 previous similar message [ 5783.103313] Lustre: Skipped 1 previous similar message [ 5784.731583] Lustre: server umount lustre-OST0000 complete [ 5788.645778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5788.661108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.677187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.250104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.254056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.905926] Lustre: server umount lustre-MDT0000 complete [ 5798.299016] LustreError: 184491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393276 with bad export cookie 11257630291329402329 [ 5798.313914] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.653269] Lustre: server umount lustre-MDT0001 complete [ 5808.526361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 5811.431697] Key type lgssc unregistered [ 5811.643449] LNet: 188071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5811.651310] LNetError: 188071:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5811.667445] LNet: Removed LNI 192.168.204.131@tcp [ 5812.403943] Key type .llcrypt unregistered [ 5812.406266] Key type ._llcrypt unregistered [ 5835.948428] Key type ._llcrypt registered [ 5835.951981] Key type .llcrypt registered [ 5836.071625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 5837.335881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5837.353806] alg: No test for adler32 (adler32-zlib) [ 5838.511713] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 5838.811503] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 5840.512542] Key type lgssc registered [ 5841.591534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5849.459879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5849.489439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.116393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.123926] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5861.346794] LustreError: 189491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.364142] LustreError: 189491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5864.799075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.326092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.872584] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5877.495627] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5884.609989] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5893.052323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.488641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.580437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5900.429184] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5908.791457] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5920.935261] Lustre: lustre-OST0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 5921.249994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5921.262501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.281027] Lustre: Skipped 1 previous similar message [ 5924.302865] Lustre: server umount lustre-OST0000 complete [ 5932.087265] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff894c05f7d000.ost_server_uuid 50 [ 5934.061259] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5948.375877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5948.388240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5948.399598] Lustre: Skipped 3 previous similar messages [ 5953.504659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5954.753393] Lustre: server umount lustre-MDT0000 complete [ 5958.629923] LustreError: 189486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5958.655224] LustreError: 189486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5958.814174] LustreError: 190226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393436 with bad export cookie 16840884769868129549 [ 5958.815797] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.826771] LustreError: 190226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5959.377711] Lustre: server umount lustre-MDT0001 complete [ 5969.613277] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 06:10:45 (1787393445) [ 5977.697843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.226672] LustreError: 193409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5978.316864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.714741] LustreError: 193410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.657921] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 5988.834449] LustreError: 193409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5994.903147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.272533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.840692] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6009.609438] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6016.053389] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6023.523315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.025992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6031.414971] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6039.793118] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6047.053871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6047.255449] Lustre: lustre-OST0001: new disk, initializing [ 6047.258749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6047.275805] Lustre: 196249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6047.375176] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6048.441601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6053.823841] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6055.048404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6055.059821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6055.180353] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6063.002452] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6065.645708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.660014] Lustre: Skipped 1 previous similar message [ 6065.664331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.673853] Lustre: Skipped 2 previous similar messages [ 6070.757302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6070.762530] Lustre: Skipped 1 previous similar message [ 6071.779276] Lustre: server umount lustre-OST0000 complete [ 6073.994798] LustreError: 196682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.013356] LustreError: 196682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6079.186509] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6079.434701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6083.910945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6084.226573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6091.242740] LustreError: 195231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.262132] LustreError: 195231:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6101.472358] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.474681] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.489192] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6101.509789] Lustre: Skipped 2 previous similar messages [ 6101.529291] Lustre: Skipped 2 previous similar messages [ 6104.575772] Lustre: server umount lustre-OST0001 complete [ 6111.720151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.733177] Lustre: Skipped 1 previous similar message [ 6115.109658] Lustre: server umount lustre-MDT0000 complete [ 6119.501588] LustreError: 196248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393597 with bad export cookie 16840884769868130613 [ 6119.519223] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6119.522332] LustreError: 196248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6119.929395] Lustre: server umount lustre-MDT0001 complete [ 6146.672695] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 06:13:43 (1787393623) [ 6153.837459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6154.125823] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6154.178364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6154.414362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.339552] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6169.069684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6169.243149] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6169.325966] Lustre: 199689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6175.393111] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6183.984158] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6190.273540] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6199.279180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.487027] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6206.834359] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6208.152586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6215.589213] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6223.127116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6223.230880] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6223.466795] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6223.472145] Lustre: Skipped 2 previous similar messages [ 6230.869876] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6232.373281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6239.774578] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6242.796956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6242.799923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.818870] Lustre: Skipped 1 previous similar message [ 6242.834518] Lustre: Skipped 3 previous similar messages [ 6248.913619] Lustre: server umount lustre-OST0000 complete [ 6250.629384] LustreError: 201543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6250.655872] LustreError: 201543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6268.386778] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6268.396021] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.408058] LustreError: Skipped 2 previous similar messages [ 6271.733886] Lustre: server umount lustre-OST0001 complete [ 6277.088653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6277.103921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6277.108042] Lustre: Skipped 5 previous similar messages [ 6282.570704] Lustre: server umount lustre-MDT0000 complete [ 6286.188212] LustreError: 201561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393764 with bad export cookie 16840884769868132027 [ 6286.192799] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6286.202846] LustreError: 201561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6286.561704] Lustre: server umount lustre-MDT0001 complete [ 6310.922195] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 06:16:27 (1787393787) [ 6328.775675] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 6340.171625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 6376.491231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6376.745121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6376.791548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6376.909834] Lustre: lustre-MDT0000: new disk, initializing [ 6376.975251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.999224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6382.249934] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6393.643809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.787714] Lustre: 207788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6393.836317] LustreError: 207799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.852190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6393.860972] LustreError: 207799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6393.872573] Lustre: Skipped 1 previous similar message [ 6393.948205] Lustre: lustre-MDT0001: new disk, initializing [ 6394.107886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6394.113917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6398.724566] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6408.372727] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6414.503505] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6421.615141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6421.895939] Lustre: lustre-OST0000: new disk, initializing [ 6421.906244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6421.914886] Lustre: 209758:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6427.926776] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6430.254413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6430.262569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6430.319616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6437.425908] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6442.496867] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6442.688525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6447.620190] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6447.856515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6450.658572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.667163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6450.677392] Lustre: Skipped 3 previous similar messages [ 6450.684066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.709496] Lustre: Skipped 3 previous similar messages [ 6456.165532] Lustre: server umount lustre-OST0000 complete [ 6460.901405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6474.207244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6474.552435] Lustre: server umount lustre-MDT0000 complete [ 6478.511751] LustreError: 207779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393956 with bad export cookie 16840884769868133812 [ 6478.527425] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.986206] Lustre: server umount lustre-MDT0001 complete [ 6488.032063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6492.759847] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6500.699303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.243693] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6512.793373] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6518.420182] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6525.933919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.000589] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6541.159837] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6546.675227] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6546.939957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6551.667778] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6551.885721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6554.087069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6554.102644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6554.126129] Lustre: Skipped 4 previous similar messages [ 6559.786747] Lustre: server umount lustre-OST0000 complete [ 6566.997362] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6571.466848] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6600.859671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.004478] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6609.455513] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.472727] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.475087] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6609.492688] Lustre: Skipped 1 previous similar message [ 6613.958232] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6619.463930] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6624.736399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6624.740328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6624.744806] LustreError: Skipped 1 previous similar message [ 6624.764652] Lustre: Skipped 13 previous similar messages [ 6630.828931] Lustre: server umount lustre-OST0000 complete [ 6640.526799] Lustre: server umount lustre-MDT0000 complete [ 6643.922120] LustreError: 211958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394121 with bad export cookie 16840884769868134694 [ 6643.925261] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.927996] LustreError: 211958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6644.197870] Lustre: server umount lustre-MDT0001 complete [ 6652.756878] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 06:22:09 (1787394129) [ 6659.195041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.621897] LustreError: 217943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6659.664115] LustreError: 217943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6659.711647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6659.718957] Lustre: Skipped 6 previous similar messages [ 6664.567954] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6673.501423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.374417] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6687.849124] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6694.134122] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6701.940612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6708.496409] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6716.863225] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6721.959649] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6722.265862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6727.197117] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6727.428783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6734.636950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.887470] Lustre: lustre-OST0001: new disk, initializing [ 6734.894544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6734.902594] Lustre: 221314:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6741.593772] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6742.127168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6742.147601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6742.265502] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6751.057704] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6756.419317] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6756.678790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6761.965559] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6762.235599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.096427] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.096487] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.107460] LustreError: Skipped 2 previous similar messages [ 6768.122051] Lustre: Skipped 4 previous similar messages [ 6770.388954] Lustre: server umount lustre-OST0001 complete [ 6777.321076] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6777.649709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6782.752665] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6783.012920] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6812.518309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.765221] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6817.776736] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6817.780915] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6817.807213] Lustre: Skipped 1 previous similar message [ 6819.147690] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6826.671740] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6829.035637] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6837.495839] Lustre: server umount lustre-OST0001 complete [ 6860.971605] Lustre: server umount lustre-OST0000 complete [ 6871.328467] Lustre: server umount lustre-MDT0000 complete [ 6874.937595] LustreError: 217925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394353 with bad export cookie 16840884769868135660 [ 6874.946761] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6874.954424] LustreError: 217925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6881.398207] Lustre: server umount lustre-MDT0001 complete [ 6905.239436] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 06:26:22 (1787394382) [ 6911.840111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.155447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6912.177271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6916.423348] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6925.669443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6925.964367] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6926.041467] Lustre: 226562:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.330311] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6943.547455] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6949.684383] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6957.319957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.600099] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6965.394173] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6973.684803] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6984.055814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.191857] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6991.330965] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6998.715348] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7000.525280] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7005.294619] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7005.421511] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7009.725142] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7009.933564] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7015.567677] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff894c09849800.ost_server_uuid 50 [ 7017.333681] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff894c09849800.ost_server_uuid in IDLE state after 0 sec [ 7018.678177] Lustre: Permanently deactivating lustre-OST0001 [ 7018.715957] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7022.928188] Lustre: Permanently reactivating lustre-OST0001 [ 7022.955373] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7022.960854] Lustre: Skipped 2 previous similar messages [ 7028.713280] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7028.726591] LustreError: Skipped 5 previous similar messages [ 7028.733887] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7028.749604] Lustre: Skipped 7 previous similar messages [ 7028.754723] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7028.757329] Lustre: Skipped 15 previous similar messages [ 7033.816218] Lustre: server umount lustre-OST0001 complete [ 7036.402098] LustreError: 228410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7036.435610] LustreError: 228410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 7043.801155] Lustre: server umount lustre-OST0000 complete [ 7054.090435] Lustre: server umount lustre-MDT0000 complete [ 7057.766058] LustreError: 226552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394535 with bad export cookie 16840884769868136948 [ 7057.767718] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7057.773471] LustreError: 226552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7058.099738] Lustre: server umount lustre-MDT0001 complete [ 7083.604550] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 06:29:20 (1787394560) [ 7091.269943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7091.517657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7091.554801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7091.565684] Lustre: Skipped 2 previous similar messages [ 7096.079188] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7104.186432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7104.237186] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7104.278769] Lustre: 233258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.114556] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7115.800406] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7123.318767] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7129.787393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.057174] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7130.082267] Lustre: Permanently deactivating lustre-OST0000 [ 7130.099948] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7136.458609] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7137.301988] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7143.220459] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7149.886173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7149.985146] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7155.813113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7155.855279] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7163.019599] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7164.871204] Lustre: Permanently reactivating lustre-OST0000 [ 7164.898677] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7164.904255] Lustre: Skipped 2 previous similar messages [ 7173.238439] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 0000000076b8ae6e, cur 1787394651 deadline 1787394645 last 1787394615 [ 7173.240814] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7173.269806] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7173.291303] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7173.362895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7185.430423] Lustre: server umount lustre-OST0001 complete [ 7194.896335] Lustre: server umount lustre-OST0000 complete [ 7205.217589] Lustre: server umount lustre-MDT0000 complete [ 7209.134921] LustreError: 233250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394687 with bad export cookie 16840884769868138607 [ 7209.144553] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7209.148152] LustreError: 233250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7209.518926] Lustre: server umount lustre-MDT0001 complete [ 7221.360568] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7224.304542] Key type lgssc unregistered [ 7224.587641] LNet: 238135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7224.599108] LNetError: 238135:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7224.622638] LNet: Removed LNI 192.168.204.131@tcp [ 7225.512170] Key type .llcrypt unregistered [ 7225.515287] Key type ._llcrypt unregistered [ 7246.507885] Key type ._llcrypt registered [ 7246.511173] Key type .llcrypt registered [ 7246.575625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7247.837070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7247.913871] alg: No test for adler32 (adler32-zlib) [ 7249.018585] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 7249.218563] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7250.872092] Key type lgssc registered [ 7252.005967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7261.238461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7261.262396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.660566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.350985] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7272.935494] LustreError: 239594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.957963] LustreError: 239594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7275.904952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7276.065210] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7276.091301] Lustre: 239582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7276.103558] Lustre: Permanently deactivating lustre-MDT0001 [ 7276.115610] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7276.373386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7277.628368] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7277.637088] LustreError: 239624:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7277.656926] LustreError: 239624:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7277.669765] LustreError: 239624:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7281.100357] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7289.188914] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7295.524036] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7303.135620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.394386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.517389] LustreError: 241436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.539636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7310.851632] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7313.900522] LustreError: 241437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7317.087525] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7319.018307] LustreError: 241668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7324.020550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.253661] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.280670] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7329.293694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7330.700635] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7338.285410] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7342.602505] Lustre: Permanently reactivating lustre-MDT0001 [ 7342.619394] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7342.624228] Lustre: Skipped 2 previous similar messages [ 7345.700879] Lustre: lustre-MDT0001: Client 4a2fc11d-e1be-422a-a85d-fe181900ed18 (at 192.168.204.31@tcp) reconnecting [ 7351.336863] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7352.758594] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7358.102902] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7358.304855] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7362.806831] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7363.079774] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7365.891378] Lustre: Permanently deactivating lustre-MDT0001 [ 7365.903164] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7365.907265] Lustre: Skipped 2 previous similar messages [ 7375.967706] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7383.018348] Lustre: server umount lustre-MDT0000 complete [ 7384.031901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7384.038346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.056183] LustreError: 240342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.069846] LustreError: 240342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7385.569749] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.578235] Lustre: Skipped 1 previous similar message [ 7386.484869] LustreError: 239575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394864 with bad export cookie 14423173157929350552 [ 7386.501547] LustreError: 239575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7386.501853] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7386.797961] Lustre: server umount lustre-MDT0001 complete [ 7392.931283] Lustre: server umount lustre-OST0000 complete [ 7404.611674] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 06:34:41 (1787394881) [ 7414.225257] Lustre: server umount lustre-OST0001 complete [ 7420.362876] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 7430.031981] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7468.447635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.731378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7468.748325] Lustre: Skipped 2 previous similar messages [ 7468.773302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7468.897637] Lustre: lustre-MDT0000: new disk, initializing [ 7469.018416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.049810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7474.164909] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7484.845440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7484.976897] Lustre: 247863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7485.057521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7485.061541] Lustre: Skipped 1 previous similar message [ 7485.138336] Lustre: lustre-MDT0001: new disk, initializing [ 7485.245788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7485.288896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7485.302448] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7489.782654] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7498.930745] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7504.814863] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7511.794935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7512.096677] Lustre: lustre-OST0000: new disk, initializing [ 7512.100950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7512.108514] Lustre: 249837:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7512.234244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.165406] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7519.832220] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7519.846681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7519.943216] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7527.877219] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7530.264359] LustreError: 249089:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7539.113273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7539.291082] Lustre: lustre-OST0001: new disk, initializing [ 7539.295249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7539.304745] Lustre: 250953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7540.320246] LustreError: 249089:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7540.362588] LustreError: 247872:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7544.631285] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7549.116089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7549.131133] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7549.194509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7550.407104] LustreError: 247872:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7554.668284] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7556.341535] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7559.656154] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7559.658417] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.667777] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7559.675254] LustreError: Skipped 1 previous similar message [ 7559.691278] Lustre: Skipped 2 previous similar messages [ 7564.607836] Lustre: server umount lustre-OST0001 complete [ 7564.783874] LustreError: 250294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.806600] LustreError: 250294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7585.254030] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.261660] LustreError: 250294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.263186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7585.263192] Lustre: Skipped 2 previous similar messages [ 7585.287078] Lustre: Skipped 1 previous similar message [ 7585.299074] LustreError: 250294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7588.697385] Lustre: server umount lustre-OST0000 complete [ 7592.928568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7592.939349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.960232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.973184] Lustre: Skipped 1 previous similar message [ 7595.491186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7595.499513] Lustre: Skipped 1 previous similar message [ 7599.270276] Lustre: server umount lustre-MDT0000 complete [ 7603.665471] LustreError: 250952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395081 with bad export cookie 14423173157929353149 [ 7603.670400] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.679778] LustreError: 250952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7610.142094] Lustre: server umount lustre-MDT0001 complete [ 7620.328870] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7623.124330] Key type lgssc unregistered [ 7623.439144] LNet: 252984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7623.450540] LNetError: 252984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7623.464087] LNet: Removed LNI 192.168.204.131@tcp [ 7624.337759] Key type .llcrypt unregistered [ 7624.340557] Key type ._llcrypt unregistered [ 7665.583549] Key type ._llcrypt registered [ 7665.586161] Key type .llcrypt registered [ 7665.686927] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7666.746159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7666.809234] alg: No test for adler32 (adler32-zlib) [ 7667.916686] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 7668.223563] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7669.959190] Key type lgssc registered [ 7671.613024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7681.413760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7681.451520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.785880] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7682.816601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7682.979215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7687.771074] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7697.137545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.267989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7697.307136] Lustre: 255194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7697.526747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7702.436493] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7709.768594] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7715.315310] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7722.278951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7722.419233] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7722.600583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7728.729681] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7732.415244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7736.747653] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7789.034378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.038127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.059617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.064397] Lustre: Skipped 1 previous similar message [ 7794.152168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.157257] Lustre: Skipped 1 previous similar message [ 7795.152277] Lustre: server umount lustre-OST0000 complete [ 7800.348707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7813.906626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7814.168555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.324755] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7819.335608] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7819.337079] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7819.359212] Lustre: Skipped 1 previous similar message [ 7822.088812] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7829.581390] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7870.433610] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.447554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7870.455330] Lustre: Skipped 1 previous similar message [ 7870.467233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7870.468711] LustreError: Skipped 1 previous similar message [ 7875.555848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.564577] Lustre: Skipped 1 previous similar message [ 7875.791718] Lustre: server umount lustre-OST0000 complete [ 7880.672565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.685687] Lustre: Skipped 1 previous similar message [ 7880.690500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.697898] Lustre: Skipped 2 previous similar messages [ 7886.260268] Lustre: server umount lustre-MDT0000 complete [ 7890.154911] LustreError: 255185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395368 with bad export cookie 18303120873409668090 [ 7890.163689] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7890.170469] LustreError: 255185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7890.662988] Lustre: server umount lustre-MDT0001 complete [ 7901.953559] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7905.287758] Key type lgssc unregistered [ 7905.674738] LNet: 260620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7905.694345] LNetError: 260620:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7906.728605] LNet: Removed LNI 192.168.204.131@tcp [ 7907.721748] Key type .llcrypt unregistered [ 7907.726872] Key type ._llcrypt unregistered [ 7935.068715] Key type ._llcrypt registered [ 7935.070243] Key type .llcrypt registered [ 7935.191208] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7936.594236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7936.700147] alg: No test for adler32 (adler32-zlib) [ 7937.773432] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 7938.021969] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7939.704227] Key type lgssc registered [ 7940.713959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7948.357453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7948.385948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7949.904358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.024596] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7959.022546] LustreError: 262039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7959.045594] LustreError: 262039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7964.006974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.129672] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7964.255611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7969.739103] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7979.073380] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7985.004625] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7992.301714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7992.725911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7994.803154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7999.737960] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8007.188215] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8034.797444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.807508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8034.816814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8039.809183] Lustre: server umount lustre-OST0000 complete [ 8045.026642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.052189] Lustre: Skipped 1 previous similar message [ 8045.060852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.073330] Lustre: Skipped 2 previous similar messages [ 8050.147716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8050.154158] Lustre: Skipped 1 previous similar message [ 8054.244565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.251228] Lustre: Skipped 1 previous similar message [ 8058.339980] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8058.756642] Lustre: server umount lustre-MDT0000 complete [ 8059.367417] LustreError: 262039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.393341] LustreError: 262039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8062.527724] LustreError: 262021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395540 with bad export cookie 9162551957576564514 [ 8062.540893] LustreError: 262021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8062.543639] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.974751] Lustre: server umount lustre-MDT0001 complete [ 8074.572167] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8077.206693] Key type lgssc unregistered [ 8077.571699] LNet: 266349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8077.585363] LNetError: 266349:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8078.629995] LNet: Removed LNI 192.168.204.131@tcp [ 8079.462186] Key type .llcrypt unregistered [ 8079.467102] Key type ._llcrypt unregistered [ 8096.259214] Key type ._llcrypt registered [ 8096.261820] Key type .llcrypt registered [ 8096.383387] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8097.363058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8097.422092] alg: No test for adler32 (adler32-zlib) [ 8098.388270] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8098.694586] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8100.351180] Key type lgssc registered [ 8101.654412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8109.826764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8109.841829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.329545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8115.715827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8121.827693] LustreError: 267574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8121.839271] LustreError: 267574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8124.043656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.410339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.975658] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8137.220764] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8146.247635] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8155.400513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8155.687860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8156.728209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8161.716435] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8169.731167] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8182.242591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.252809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.266861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8183.225347] Lustre: server umount lustre-OST0000 complete [ 8189.409614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8189.416810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8189.428976] Lustre: Skipped 1 previous similar message [ 8189.446115] Lustre: Skipped 1 previous similar message [ 8193.447280] Lustre: server umount lustre-MDT0000 complete [ 8194.530566] LustreError: 267573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8194.557865] LustreError: 267573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8197.684486] LustreError: 268306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395675 with bad export cookie 11487694352921777644 [ 8197.691207] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.711948] LustreError: 268306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8207.963547] Lustre: server umount lustre-MDT0001 complete [ 8218.415702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8220.894617] Key type lgssc unregistered [ 8221.182801] LNet: 271249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8221.190169] LNetError: 271249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8221.213725] LNet: Removed LNI 192.168.204.131@tcp [ 8222.130779] Key type .llcrypt unregistered [ 8222.134621] Key type ._llcrypt unregistered [ 8247.750457] Key type ._llcrypt registered [ 8247.754441] Key type .llcrypt registered [ 8247.889947] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8249.141016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8249.272431] alg: No test for adler32 (adler32-zlib) [ 8250.376528] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8250.669883] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8252.351192] Key type lgssc registered [ 8253.746204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8262.114758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8262.150739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.596510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.400719] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8273.892254] LustreError: 272667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.911873] LustreError: 272667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8277.847475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.194240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8282.953565] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8291.019181] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8298.693262] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8308.123647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.512223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.144662] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8323.240223] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8333.103941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8358.883421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.891155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.896539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8358.913159] Lustre: Skipped 1 previous similar message [ 8364.008695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.020781] Lustre: Skipped 2 previous similar messages [ 8364.498661] Lustre: server umount lustre-OST0000 complete [ 8369.123092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8369.143070] Lustre: Skipped 1 previous similar message [ 8369.151354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.244433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.247838] Lustre: Skipped 2 previous similar messages [ 8374.659127] Lustre: server umount lustre-MDT0000 complete [ 8378.338164] LustreError: 272646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395856 with bad export cookie 8388508332820334371 [ 8378.347862] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8378.354573] LustreError: 272646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8378.707963] Lustre: server umount lustre-MDT0001 complete [ 8388.428827] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8391.176882] Key type lgssc unregistered [ 8391.504474] LNet: 276993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8391.514992] LNetError: 276993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8391.541223] LNet: Removed LNI 192.168.204.131@tcp [ 8392.159112] Key type .llcrypt unregistered [ 8392.164274] Key type ._llcrypt unregistered [ 8410.528460] Key type ._llcrypt registered [ 8410.531475] Key type .llcrypt registered [ 8410.672928] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8411.779714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8411.872799] alg: No test for adler32 (adler32-zlib) [ 8413.134475] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8413.449303] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8415.151165] Key type lgssc registered [ 8416.422312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8425.666904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8425.686553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.338810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.568996] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8437.730862] LustreError: 278232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.762995] LustreError: 278232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8442.521433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.978047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8448.559894] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8457.257920] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8463.321430] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8471.585184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.077525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.148247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8480.158468] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8488.704942] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8502.761537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8502.764603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8502.781321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8502.790971] Lustre: Skipped 1 previous similar message [ 8502.802130] Lustre: Skipped 1 previous similar message [ 8506.662646] Lustre: server umount lustre-OST0000 complete [ 8512.995599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8513.009036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.012543] Lustre: Skipped 1 previous similar message [ 8513.024557] Lustre: Skipped 1 previous similar message [ 8518.115609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.121415] Lustre: Skipped 1 previous similar message [ 8522.228109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.237021] Lustre: Skipped 1 previous similar message [ 8525.279263] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8525.589958] Lustre: server umount lustre-MDT0000 complete [ 8527.338935] LustreError: 278215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.372616] LustreError: 278215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8529.903075] LustreError: 278198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396007 with bad export cookie 8378853080031672639 [ 8529.913644] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8529.922984] LustreError: 278198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8530.381444] Lustre: server umount lustre-MDT0001 complete [ 8543.163670] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8546.402497] Key type lgssc unregistered [ 8546.776711] LNet: 281934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8546.784416] LNetError: 281934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8546.808266] LNet: Removed LNI 192.168.204.131@tcp [ 8547.806940] Key type .llcrypt unregistered [ 8547.813693] Key type ._llcrypt unregistered [ 8561.918820] Key type ._llcrypt registered [ 8561.922215] Key type .llcrypt registered [ 8562.023123] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8581.699523] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8600.959919] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8601.906982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8601.974091] alg: No test for adler32 (adler32-zlib) [ 8603.085397] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8603.350681] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8604.991183] Key type lgssc registered [ 8606.172617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8649.261724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8649.294372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.530499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8650.563955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8650.661393] Lustre: lustre-MDT0000: new disk, initializing [ 8650.743988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.769623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8655.543068] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8668.007251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.153038] Lustre: 285501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8668.240298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8668.245560] Lustre: Skipped 1 previous similar message [ 8668.331203] Lustre: lustre-MDT0001: new disk, initializing [ 8668.460194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8668.508286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8668.533768] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8673.498494] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8683.416540] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8689.503302] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8697.614044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.066453] Lustre: lustre-OST0000: new disk, initializing [ 8698.072527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8698.078390] Lustre: 287471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8698.188795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.605269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8704.627803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8704.806271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8705.587632] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8715.939778] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8720.888726] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8721.074811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8725.712812] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8725.925671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8730.594555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8730.596684] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.607800] LustreError: Skipped 1 previous similar message [ 8730.616542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8735.722641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8735.736113] Lustre: Skipped 1 previous similar message [ 8740.834529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8740.844592] Lustre: Skipped 1 previous similar message [ 8742.367175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8742.581691] Lustre: server umount lustre-OST0000 complete [ 8750.562578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8750.574268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.594146] Lustre: Skipped 1 previous similar message [ 8750.604662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.192434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8756.205439] Lustre: Skipped 3 previous similar messages [ 8761.315126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8761.529144] Lustre: server umount lustre-MDT0000 complete [ 8765.520349] LustreError: 285495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396243 with bad export cookie 3226220979799489315 [ 8765.525965] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.540762] LustreError: 285495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8766.098586] Lustre: server umount lustre-MDT0001 complete [ 8776.388893] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 06:57:32 (1787396252) [ 8784.848315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.466197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8790.504165] LustreError: 289889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.527296] LustreError: 289889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8790.593222] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8795.617285] LustreError: 289890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8800.019203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.404771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8805.390680] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8814.738995] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8821.678288] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8829.403779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.849816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8836.788829] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8844.483808] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8855.533343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8855.535688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.549263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8855.549274] Lustre: Skipped 2 previous similar messages [ 8855.579839] Lustre: Skipped 2 previous similar messages [ 8860.257433] Lustre: server umount lustre-OST0000 complete [ 8865.763601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.785338] Lustre: Skipped 1 previous similar message [ 8870.761454] Lustre: server umount lustre-MDT0000 complete [ 8870.880578] LustreError: 289890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.909103] LustreError: 289890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8874.987267] LustreError: 289884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8875.010457] LustreError: 289884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8875.026299] LustreError: 290623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396353 with bad export cookie 3226220979799490197 [ 8875.043935] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8875.449733] Lustre: server umount lustre-MDT0001 complete [ 8887.370683] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8890.145763] Key type lgssc unregistered [ 8890.451624] LNet: 293422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8890.458052] LNetError: 293422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8890.468799] LNet: Removed LNI 192.168.204.131@tcp [ 8891.404709] Key type .llcrypt unregistered [ 8891.409822] Key type ._llcrypt unregistered [ 8921.661774] Key type ._llcrypt registered [ 8921.664652] Key type .llcrypt registered [ 8921.802466] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8939.588832] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8940.861335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8940.892240] alg: No test for adler32 (adler32-zlib) [ 8942.037414] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 8942.410589] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8944.207215] Key type lgssc registered [ 8945.786639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9002.213304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9002.263352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.599281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9003.647646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9003.761204] Lustre: lustre-MDT0000: new disk, initializing [ 9003.889060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.919092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9009.822381] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9021.823914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.041867] Lustre: 297267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9022.097103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9022.101199] Lustre: Skipped 1 previous similar message [ 9022.168888] Lustre: lustre-MDT0001: new disk, initializing [ 9022.269175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.313132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9022.326293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9027.828951] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9038.876889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9046.283660] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9055.178886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9055.432243] Lustre: lustre-OST2710: new disk, initializing [ 9055.447235] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9055.454223] Lustre: 299236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9055.553475] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9062.141048] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9062.568236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9062.578057] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9062.697902] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9072.922967] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9080.308286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.515779] Lustre: lustre-OST03e8: new disk, initializing [ 9080.525492] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9080.547462] Lustre: 300299:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9080.678789] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9087.112370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9087.128971] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9087.257509] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9087.559801] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9095.618852] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9103.190204] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9103.405881] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9108.045582] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9108.183318] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9111.991519] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9112.188785] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9116.660438] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9116.850894] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9123.307042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9123.318412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.330748] Lustre: Skipped 2 previous similar messages [ 9123.561751] Lustre: server umount lustre-MDT0000 complete [ 9127.474543] LustreError: 297259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396605 with bad export cookie 14539912401151274977 [ 9127.481818] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9127.484147] LustreError: 297259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9127.867107] Lustre: server umount lustre-MDT0001 complete [ 9142.547665] Lustre: server umount lustre-OST2710 complete [ 9144.799455] Lustre: 295056:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396606/real 1787396606] req@00000000e3f9de42 x1874220988566656/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787396622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9144.830901] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9144.858889] Lustre: Skipped 3 previous similar messages [ 9146.418686] Lustre: server umount lustre-OST03e8 complete [ 9166.130643] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9176.236527] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9210.943529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.131648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9211.159880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9211.255418] Lustre: lustre-MDT0000: new disk, initializing [ 9211.336470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9211.361979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9215.969735] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9226.940109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9227.100112] Lustre: 305547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9227.254921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9227.258181] Lustre: Skipped 1 previous similar message [ 9227.387737] Lustre: lustre-MDT0001: new disk, initializing [ 9227.511856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9227.576604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9227.600171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9233.340844] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9244.532876] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9251.332475] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9258.405224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9258.760746] Lustre: lustre-OST0000: new disk, initializing [ 9258.769541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9258.782573] Lustre: 307517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9258.927468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9265.831036] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9268.857345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9268.865429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9268.981145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9275.420388] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9280.283837] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9280.496848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9285.350342] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9285.555892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9289.697918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9289.701190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9289.705739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9289.705747] Lustre: Skipped 1 previous similar message [ 9289.719169] Lustre: Skipped 1 previous similar message [ 9289.742433] LustreError: Skipped 1 previous similar message [ 9293.749349] Lustre: server umount lustre-OST0000 complete [ 9299.433305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9299.441694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9299.459806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9299.469326] Lustre: Skipped 1 previous similar message [ 9304.479450] Lustre: server umount lustre-MDT0000 complete [ 9308.033323] LustreError: 306342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396786 with bad export cookie 14539912401151276664 [ 9308.039730] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9308.044395] LustreError: 306342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9308.436989] Lustre: server umount lustre-MDT0001 complete [ 9318.216520] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 07:06:34 (1787396794) [ 9319.956312] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9321.744213] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 07:06:38 (1787396798) [ 9325.489633] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9351.492333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.746438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9351.789558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9351.980274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9356.101903] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9363.813435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.931564] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9363.956311] Lustre: 311376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9368.018266] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9389.501095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9389.767688] Lustre: Denying initial registration attempt from nid 192.168.204.131@tcp, specified as failover [ 9389.780733] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9389.794446] LustreError: 312830:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9389.801715] LustreError: 312830:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9389.818762] LustreError: 312830:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9389.969562] Lustre: server umount lustre-OST0000 complete [ 9389.971933] LustreError: 312830:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9400.293105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.307499] Lustre: Skipped 1 previous similar message [ 9400.307984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9400.312714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9400.312721] Lustre: Skipped 1 previous similar message [ 9403.556993] Lustre: server umount lustre-MDT0000 complete [ 9405.417933] LustreError: 311388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.449421] LustreError: 311388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9407.505269] LustreError: 311367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396885 with bad export cookie 14539912401151277546 [ 9407.506230] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9407.511933] LustreError: 311367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9407.995459] Lustre: server umount lustre-MDT0001 complete [ 9419.637349] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9422.437497] Key type lgssc unregistered [ 9422.720617] LNet: 313952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9422.727055] LNetError: 313952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9422.741382] LNet: Removed LNI 192.168.204.131@tcp [ 9423.510054] Key type .llcrypt unregistered [ 9423.514819] Key type ._llcrypt unregistered [ 9434.611750] Key type ._llcrypt registered [ 9434.614597] Key type .llcrypt registered [ 9434.700295] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9435.763324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9435.781903] alg: No test for adler32 (adler32-zlib) [ 9436.881145] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 9437.130127] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9438.824969] Key type lgssc registered [ 9440.015885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9487.206815] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9494.642829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9494.665904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.972845] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9495.994366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9496.164723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9500.084993] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9507.899451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.013729] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9508.057435] Lustre: 316716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9508.377915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9512.593209] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9519.784374] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9524.709945] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9544.056170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9544.333443] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9544.482908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.660474] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9558.677690] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9561.576847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9561.581372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.596409] LustreError: Skipped 1 previous similar message [ 9561.601105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9561.609797] Lustre: Skipped 1 previous similar message [ 9561.633344] Lustre: Skipped 1 previous similar message [ 9566.689674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9566.704537] Lustre: Skipped 1 previous similar message [ 9571.817527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9571.820111] Lustre: Skipped 1 previous similar message [ 9575.391154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9575.614389] Lustre: server umount lustre-OST0000 complete [ 9580.005092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9580.014896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9580.050314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9582.049535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9587.169469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9587.173845] Lustre: Skipped 3 previous similar messages [ 9594.335255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9594.504919] Lustre: server umount lustre-MDT0000 complete [ 9598.182121] LustreError: 317457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397076 with bad export cookie 15493475938400820924 [ 9598.190690] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9598.194571] LustreError: 317457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9598.593812] Lustre: server umount lustre-MDT0001 complete [ 9609.322468] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9612.161764] Key type lgssc unregistered [ 9612.504421] LNet: 320291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9612.518352] LNetError: 320291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9612.534140] LNet: Removed LNI 192.168.204.131@tcp [ 9613.456222] Key type .llcrypt unregistered [ 9613.458300] Key type ._llcrypt unregistered [ 9637.243215] Key type ._llcrypt registered [ 9637.251961] Key type .llcrypt registered [ 9637.385611] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9638.763985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9638.864423] alg: No test for adler32 (adler32-zlib) [ 9639.946383] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 9640.315611] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9641.999210] Key type lgssc registered [ 9643.126113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9651.139319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9651.152090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9652.651537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9656.906314] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9662.945312] LustreError: 321709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.961337] LustreError: 321709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9665.735078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.130370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9669.782762] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9677.537256] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9683.469410] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9692.600814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9692.993635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9698.794530] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9706.374839] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9721.827042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9721.839379] Lustre: Skipped 1 previous similar message [ 9721.845646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9721.849637] Lustre: Skipped 2 previous similar messages [ 9723.556938] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [ 9726.338442] Lustre: server umount lustre-MDT0000 complete [ 9726.955511] LustreError: 321704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.968886] LustreError: 321704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9728.643157] LustreError: 323838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.382316] LustreError: 321690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397207 with bad export cookie 8148401318807911630 [ 9729.387558] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.396418] LustreError: 321690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9732.064047] LustreError: 321706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9732.070270] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.102495] Lustre: Skipped 1 previous similar message [ 9732.106938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9734.113145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9734.119809] Lustre: Skipped 1 previous similar message [ 9735.739812] Lustre: server umount lustre-MDT0001 complete [ 9741.828605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9750.276048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.593335] LustreError: 321019:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e003c96f x1874221720449152/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9754.883397] LustreError: 323542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9754.900492] LustreError: 323542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9754.968153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9755.030963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9758.417973] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9764.196776] LustreError: 325325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.199649] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9764.215320] LustreError: 325325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9765.683134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.077290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.609285] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9771.506344] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9777.726508] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9779.201674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9785.721493] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9787.795139] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9796.065810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9796.073268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9796.093845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9797.088179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9799.702687] Lustre: server umount lustre-OST0000 complete [ 9807.336392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9807.346864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9807.353401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9807.353409] Lustre: Skipped 2 previous similar messages [ 9807.379438] Lustre: Skipped 2 previous similar messages [ 9809.619359] Lustre: server umount lustre-MDT0000 complete [ 9812.452344] LustreError: 325331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9812.474483] LustreError: 325331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9814.205209] LustreError: 324516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397292 with bad export cookie 8148401318807919477 [ 9814.208535] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.224126] LustreError: 324516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9814.504234] Lustre: server umount lustre-MDT0001 complete [ 9826.463711] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9829.277021] Key type lgssc unregistered [ 9829.603821] LNet: 327996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9829.612964] LNetError: 327996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9829.630279] LNet: Removed LNI 192.168.204.131@tcp [ 9830.477930] Key type .llcrypt unregistered [ 9830.480110] Key type ._llcrypt unregistered [ 9855.376670] Key type ._llcrypt registered [ 9855.380693] Key type .llcrypt registered [ 9855.476621] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9856.777647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9856.813309] alg: No test for adler32 (adler32-zlib) [ 9857.957396] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [ 9858.250408] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9859.945075] Key type lgssc registered [ 9860.901407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9868.807573] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9868.844763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.255057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.722357] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9880.552792] LustreError: 329413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.565748] LustreError: 329413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9883.482398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.820994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9887.338453] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9894.070241] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9901.548254] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9905.633706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9905.652806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.774889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9908.790428] Lustre: Skipped 2 previous similar messages [ 9911.550243] Lustre: server umount lustre-MDT0000 complete [ 9913.829433] LustreError: 329408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9913.857114] LustreError: 329408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9915.464439] LustreError: 329394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397393 with bad export cookie 2392316999746888294 [ 9915.475896] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9915.484031] LustreError: 329394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9915.731978] Lustre: server umount lustre-MDT0001 complete [ 9931.900949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9932.273087] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9932.300633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9932.460202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9937.777922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9947.037289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9947.167465] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9947.206907] Lustre: 332001:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9947.556446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9952.567479] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9960.636825] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9965.852997] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9975.468704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9975.733358] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9975.739875] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9975.758819] LustreError: 333886:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9975.763327] LustreError: 333886:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9975.767793] LustreError: 333886:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9975.824964] Lustre: server umount lustre-OST0000 complete [ 9975.827309] LustreError: 333886:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9987.243978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9987.565327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9987.805498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9993.762845] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9995.999516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10003.297835] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10013.091493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.264894] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10013.381607] Lustre: lustre-OST0001: new disk, initializing [10013.388262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10013.399849] Lustre: 335429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10013.554276] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10021.107056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10021.125813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10021.267730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10021.841585] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10030.007549] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10036.704817] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10036.711555] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.725400] LustreError: Skipped 1 previous similar message [10036.737971] Lustre: Skipped 1 previous similar message [10036.753217] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10036.764277] Lustre: Skipped 1 previous similar message [10039.302596] Lustre: server umount lustre-OST0001 complete [10041.834795] LustreError: 334353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10041.877993] LustreError: 334353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10046.949279] LustreError: 334354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10046.950531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10046.954422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10046.955277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10046.989315] LustreError: 334354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10049.986195] Lustre: server umount lustre-OST0000 complete [10055.144878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10055.153718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10055.163121] Lustre: Skipped 1 previous similar message [10055.169761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10055.174742] Lustre: Skipped 1 previous similar message [10059.897738] Lustre: server umount lustre-MDT0000 complete [10062.307624] LustreError: 332863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10062.337199] LustreError: 332863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10063.562791] LustreError: 331991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397541 with bad export cookie 2392316999746888847 [10063.575841] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10063.576609] LustreError: 331991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10063.927372] Lustre: server umount lustre-MDT0001 complete [10074.586500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [10077.774217] Key type lgssc unregistered [10078.064834] LNet: 337385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10078.074107] LNetError: 337385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10078.090140] LNet: Removed LNI 192.168.204.131@tcp [10079.075957] Key type .llcrypt unregistered [10079.079958] Key type ._llcrypt unregistered [10160.634434] Key type ._llcrypt registered [10160.637425] Key type .llcrypt registered [10160.770170] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10177.368037] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10178.474698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10178.648414] alg: No test for adler32 (adler32-zlib) [10179.838945] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [10180.206182] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [10181.896604] Key type lgssc registered [10183.564502] Lustre: Echo OBD driver; http://www.lustre.org/ [10224.703190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10224.747589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10226.096777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10226.157657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10226.263585] Lustre: lustre-MDT0000: new disk, initializing [10226.353550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10226.375528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10231.544727] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10242.434225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10242.583955] Lustre: 342325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10242.637834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10242.641499] Lustre: Skipped 1 previous similar message [10242.754174] Lustre: lustre-MDT0001: new disk, initializing [10242.858323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10242.898407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10242.919343] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10248.259671] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10257.651791] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10264.393348] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10273.235305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10273.558137] Lustre: lustre-OST0000: new disk, initializing [10273.561351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10273.567424] Lustre: 344294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10273.687332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10280.549171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10280.568331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10280.701602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10281.124636] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10291.741567] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10297.350607] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10297.711824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10303.291158] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10303.627395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10306.533726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10306.540103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.548194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.560826] Lustre: Skipped 1 previous similar message [10311.655885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10311.672990] Lustre: Skipped 1 previous similar message [10312.431969] Lustre: server umount lustre-OST0000 complete [10319.842666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10319.861287] LustreError: Skipped 1 previous similar message [10319.874894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10319.907418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.891087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10321.910192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10321.917661] Lustre: Skipped 1 previous similar message [10323.429344] Lustre: server umount lustre-MDT0000 complete [10328.120790] LustreError: 342315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397806 with bad export cookie 16413218794768365019 [10328.121823] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.135777] LustreError: 342315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10328.611176] Lustre: server umount lustre-MDT0001 complete [10339.248310] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 07:23:35 (1787397815) [10347.684495] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 07:23:44 (1787397824) [10357.196899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.739103] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10357.773078] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10357.914863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10363.360404] LustreError: 347052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.689920] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10367.462600] LustreError: 347051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10372.584655] LustreError: 347052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10373.155291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10373.462779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10379.417497] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10387.913201] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10393.540559] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10402.046247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10402.219266] LustreError: 348873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.617822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10410.077552] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10419.068342] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10428.091537] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10430.291327] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10432.096185] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10433.944884] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10435.967087] Lustre: Failing over lustre-MDT0000 [10436.372027] Lustre: server umount lustre-MDT0000 complete [10437.281330] LustreError: 347048:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.626946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.635326] Lustre: Skipped 2 previous similar messages [10453.703123] LustreError: 349593:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.740153] LustreError: 349593:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [10453.987434] Lustre: 340549:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397916/real 1787397916] req@0000000057847a9c x1874222286709504/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787397932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10454.012304] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10455.846645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.239689] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe3c777886aeefd6e [10464.250810] Lustre: MGC192.168.204.131@tcp: Connection restored to 0@lo (at 0@lo) [10464.493319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10464.565966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10467.972717] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10469.498275] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10469.890863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10469.935402] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10470.006642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10480.432656] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10482.500266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10486.808522] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10489.147571] Lustre: Failing over lustre-MDT0000 [10489.549788] Lustre: server umount lustre-MDT0000 complete [10490.341200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.341871] LustreError: 347047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.348780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10490.355685] Lustre: Skipped 2 previous similar messages [10490.395881] LustreError: 347047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [10506.720283] Lustre: 340547:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397968/real 1787397968] req@00000000698c1f4e x1874222286729984/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787397984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10506.809887] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10511.762908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10515.937798] LustreError: 340545:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008b53136c x1874222286735360/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10516.399972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10516.481098] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10519.182959] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10521.592536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10521.602676] Lustre: Skipped 2 previous similar messages [10521.664902] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10521.728085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10521.984580] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10532.583522] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10534.684516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10542.054562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10542.058614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10542.076635] Lustre: Skipped 2 previous similar messages [10542.081561] Lustre: Skipped 1 previous similar message [10547.517057] Lustre: server umount lustre-MDT0000 complete [10551.866217] LustreError: 347031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398029 with bad export cookie 16413218794768367735 [10551.884339] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10551.885217] LustreError: 347031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10552.231883] Lustre: server umount lustre-MDT0001 complete [10567.386734] Lustre: server umount lustre-OST0000 complete [10587.220494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10587.831517] LustreError: 353396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.856432] LustreError: 353396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [10587.911240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.310459] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10604.577172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.659437] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10619.828521] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10627.135498] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10635.452299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10642.916263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10652.606970] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10657.398607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10664.930827] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10664.949196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10664.981102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10664.990457] Lustre: Skipped 4 previous similar messages [10678.239228] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10678.522391] Lustre: server umount lustre-OST0000 complete [10686.437653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.447650] Lustre: Skipped 7 previous similar messages [10689.415125] Lustre: server umount lustre-MDT0000 complete [10693.764192] LustreError: 353377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398171 with bad export cookie 16413218794768368484 [10693.776234] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10694.105605] Lustre: server umount lustre-MDT0001 complete [10705.709892] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [10708.969778] Key type lgssc unregistered [10709.324397] LNet: 356931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10709.332561] LNetError: 356931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10709.355976] LNet: Removed LNI 192.168.204.131@tcp [10710.432295] Key type .llcrypt unregistered [10710.437478] Key type ._llcrypt unregistered [10742.053242] Key type ._llcrypt registered [10742.055453] Key type .llcrypt registered [10742.158133] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10743.454239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10743.466360] alg: No test for adler32 (adler32-zlib) [10744.768650] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [10745.110991] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [10746.856932] Key type lgssc registered [10748.367470] Lustre: Echo OBD driver; http://www.lustre.org/ [10758.900312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10758.945134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10760.604639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10766.184145] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10770.912955] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.934833] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10776.035907] LustreError: 358450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.746438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.331071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10783.519562] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10792.709093] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10800.202360] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10808.331083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10808.614614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10816.934203] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10826.833820] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10830.172559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10845.670667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.678223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.706458] Lustre: Skipped 1 previous similar message [10845.721537] Lustre: Skipped 2 previous similar messages [10850.807155] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.825368] LustreError: 358451:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10850.974921] Lustre: server umount lustre-MDT0000 complete [10855.444520] LustreError: 358432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398333 with bad export cookie 10364872811974026286 [10855.453130] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.472345] LustreError: 358432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10855.890394] Lustre: server umount lustre-MDT0001 complete [10870.585253] Lustre: server umount lustre-OST0000 complete [10887.660256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.212278] LustreError: 362188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10888.292055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10893.697195] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10893.802038] LustreError: 362189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.442610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10903.785812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.957959] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10918.201951] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10925.033643] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10932.781654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10933.482061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10934.506365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10942.179943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10951.472859] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10965.475301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10965.490698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10965.501518] Lustre: Skipped 1 previous similar message [10965.513589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10969.576566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.577604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10969.601104] Lustre: Skipped 1 previous similar message [10969.611179] Lustre: Skipped 1 previous similar message [10971.247217] Lustre: server umount lustre-MDT0000 complete [10974.693978] LustreError: 362183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10974.713333] LustreError: 362183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10975.366935] LustreError: 362169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398453 with bad export cookie 10364872811974027399 [10975.378054] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10975.388451] LustreError: 362169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10976.030551] Lustre: server umount lustre-MDT0001 complete [10990.751903] Lustre: server umount lustre-OST0000 complete [11008.719300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.222379] LustreError: 366101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.323983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11014.731845] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11026.160336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.131862] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11039.542602] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11045.976340] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11053.661260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.100977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11054.110959] Lustre: Skipped 1 previous similar message [11055.193149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11061.626622] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11070.162659] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11075.556838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11075.565575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11075.580214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11075.589154] Lustre: Skipped 1 previous similar message [11077.091793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11077.437242] Lustre: server umount lustre-OST0000 complete [11082.209742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.220445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11082.229564] Lustre: Skipped 2 previous similar messages [11082.241584] Lustre: Skipped 2 previous similar messages [11088.317420] Lustre: server umount lustre-MDT0000 complete [11092.448981] LustreError: 366101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.475687] LustreError: 366101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11092.854880] LustreError: 366082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398570 with bad export cookie 10364872811974028421 [11092.862608] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11092.878139] LustreError: 366082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11093.386986] Lustre: server umount lustre-MDT0001 complete [11106.175723] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11109.048970] Key type lgssc unregistered [11109.369835] LNet: 369635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11109.375112] LNetError: 369635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11109.397276] LNet: Removed LNI 192.168.204.131@tcp [11110.632587] Key type .llcrypt unregistered [11110.633841] Key type ._llcrypt unregistered [11143.421866] Key type ._llcrypt registered [11143.423279] Key type .llcrypt registered [11143.591463] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11145.597502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11145.653814] alg: No test for adler32 (adler32-zlib) [11146.817794] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11147.170032] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11148.855209] Key type lgssc registered [11150.470439] Lustre: Echo OBD driver; http://www.lustre.org/ [11159.644959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11159.662536] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11159.671910] LustreError: 371078:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11159.714257] LustreError: 371078:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11159.722373] LustreError: 371078:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11159.732810] LustreError: 371078:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11159.750104] LustreError: 371078:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11168.466983] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11168.473687] LustreError: 371476:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11168.505921] LustreError: 371476:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11168.516166] LustreError: 371476:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11168.528621] LustreError: 371476:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11168.536974] LustreError: 371476:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11192.596289] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11195.886860] Key type lgssc unregistered [11196.196649] LNet: 372480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11196.204359] LNetError: 372480:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11196.220890] LNet: Removed LNI 192.168.204.131@tcp [11197.421316] Key type .llcrypt unregistered [11197.423925] Key type ._llcrypt unregistered [11219.856685] Key type ._llcrypt registered [11219.859112] Key type .llcrypt registered [11220.017696] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [11237.123861] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11237.858343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11237.986302] alg: No test for adler32 (adler32-zlib) [11239.032600] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11239.316848] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11241.049077] Key type lgssc registered [11242.271833] Lustre: Echo OBD driver; http://www.lustre.org/ [11283.373804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11283.417276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.712981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11284.739654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11284.802641] Lustre: lustre-MDT0000: new disk, initializing [11284.881212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.900926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11289.608652] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11301.480517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.569161] Lustre: 375640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11301.673451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11301.678578] Lustre: Skipped 1 previous similar message [11301.847442] Lustre: lustre-MDT0001: new disk, initializing [11301.951308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11302.016683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11302.040224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11307.145664] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11317.299948] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11324.689719] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11332.343026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11332.692729] Lustre: lustre-OST0000: new disk, initializing [11332.696805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11332.710893] Lustre: 377607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11332.868492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11339.521273] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11341.387246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11341.398637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11341.508259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11349.871295] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11355.145978] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11355.467698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11361.136840] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11361.501930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11367.397260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11367.405066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11367.409501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11367.411244] Lustre: Skipped 1 previous similar message [11367.425551] LustreError: Skipped 1 previous similar message [11372.521019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11372.528584] Lustre: Skipped 2 previous similar messages [11377.633161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11377.646541] Lustre: Skipped 1 previous similar message [11378.145401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11378.429405] Lustre: server umount lustre-OST0000 complete [11382.754535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11382.760489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11382.774956] Lustre: Skipped 1 previous similar message [11382.799873] Lustre: Skipped 1 previous similar message [11387.890325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11387.902206] Lustre: Skipped 1 previous similar message [11389.040909] Lustre: server umount lustre-MDT0000 complete [11393.175272] LustreError: 375632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398871 with bad export cookie 4196011778813538331 [11393.188748] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11393.194483] LustreError: 375632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11393.647775] Lustre: server umount lustre-MDT0001 complete [11403.293482] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11405.229235] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 07:41:21 (1787398881) [11412.447158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11413.031572] LustreError: 380069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11413.051954] LustreError: 380069:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11413.112896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.260373] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11418.596227] LustreError: 380070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11423.714815] LustreError: 380069:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11428.099650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11428.524299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11434.155942] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11443.865074] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11452.811070] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11462.557974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.021719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11472.464995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11481.777358] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11490.973898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11491.276587] Lustre: lustre-OST0001: new disk, initializing [11491.280702] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11491.291552] Lustre: 382910:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11491.409350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11498.945512] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11499.668468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11499.692872] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11499.933633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11510.576188] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11515.378037] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11515.394737] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.409220] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11515.421311] Lustre: Skipped 1 previous similar message [11521.253703] Lustre: server umount lustre-OST0001 complete [11522.688228] LustreError: 381894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.729537] LustreError: 381894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.764508] LustreError: 381894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11539.936700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11539.952307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.962346] LustreError: Skipped 2 previous similar messages [11539.964483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11539.964493] Lustre: Skipped 3 previous similar messages [11540.003419] Lustre: Skipped 2 previous similar messages [11540.962520] LustreError: 381894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.998663] LustreError: 381894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11545.552508] Lustre: server umount lustre-OST0000 complete [11551.202773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.217955] Lustre: Skipped 1 previous similar message [11551.223357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11556.701556] Lustre: server umount lustre-MDT0000 complete [11560.824853] LustreError: 380050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399038 with bad export cookie 4196011778813539213 [11560.837105] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11560.850508] LustreError: 380050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11575.263575] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11575.501529] Lustre: server umount lustre-MDT0001 complete [11589.534799] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11593.046381] Key type lgssc unregistered [11593.351671] LNet: 384903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11593.368245] LNetError: 384903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11593.389551] LNet: Removed LNI 192.168.204.131@tcp [11594.478540] Key type .llcrypt unregistered [11594.481727] Key type ._llcrypt unregistered [11654.237519] Key type ._llcrypt registered [11654.243511] Key type .llcrypt registered [11654.392914] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11655.830155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11655.899870] alg: No test for adler32 (adler32-zlib) [11657.095761] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [11657.465071] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11659.183167] Key type lgssc registered [11660.328539] Lustre: Echo OBD driver; http://www.lustre.org/ [11668.137085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11668.161084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11669.482841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11669.500596] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11669.690647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11674.654417] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11685.516411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11685.611148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11685.655734] Lustre: 387703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11686.005205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11691.583796] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11700.596508] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11707.139360] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11711.457853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11711.465101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.483624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11711.971227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.995134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11715.922902] Lustre: server umount lustre-MDT0000 complete [11719.978804] LustreError: 387696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399198 with bad export cookie 2117934563435036852 [11719.987069] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11719.987601] LustreError: 387696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11720.347806] Lustre: server umount lustre-MDT0001 complete [11730.733618] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 07:46:47 (1787399207) [11737.481028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.913293] LustreError: 390214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.925626] LustreError: 390214:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11737.988779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11743.200389] LustreError: 390215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.428753] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11748.323394] LustreError: 390214:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.459247] LustreError: 390215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.527419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11753.963035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11760.779537] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11769.862449] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11776.872209] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11783.936092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11784.124876] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11784.330803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11791.394507] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11799.563940] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11809.071433] Lustre: Permanently deactivating lustre-OST0000 [11809.104055] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11819.033450] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11821.555480] LustreError: 393254:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11821.568367] LustreError: 393254:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11821.581744] LustreError: 393254:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11825.253357] LustreError: 393304:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11825.260480] LustreError: 393304:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11833.390643] Lustre: server umount lustre-OST0000 complete [11836.315500] LustreError: 393553:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11836.329589] LustreError: 393553:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11838.945809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.951737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11838.961141] Lustre: Skipped 1 previous similar message [11838.980556] Lustre: Skipped 1 previous similar message [11844.064805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11844.079711] Lustre: Skipped 1 previous similar message [11848.244671] Lustre: server umount lustre-MDT0000 complete [11849.187091] LustreError: 390215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11849.206412] LustreError: 390215:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11852.397716] LustreError: 390194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399330 with bad export cookie 2117934563435037433 [11852.400780] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11852.404746] LustreError: 390194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11862.246963] Lustre: server umount lustre-MDT0001 complete [11871.800926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11876.247568] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11879.035779] LustreError: 394712:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11885.009537] Lustre: 395051:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.131@tcp [11887.214394] Lustre: server umount MGS complete [11900.682224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11901.431122] LustreError: 395776:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11901.468399] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11901.474893] Lustre: Skipped 1 previous similar message [11901.536041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11906.858930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11917.453740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.849879] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11917.950128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11923.060727] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11931.329513] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11937.569638] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11939.613725] Lustre: Permanently reactivating lustre-OST0000 [11939.625231] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11939.630210] Lustre: Skipped 2 previous similar messages [11969.505972] LustreError: 397605:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11978.778829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11979.161181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11986.245921] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11995.038401] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12002.244263] Lustre: server umount lustre-OST0000 complete [12009.955536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.959206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.972649] Lustre: Skipped 1 previous similar message [12009.989568] Lustre: Skipped 1 previous similar message [12016.171876] Lustre: server umount lustre-MDT0000 complete [12020.193034] Lustre: 387025:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787399482/real 1787399482] req@00000000c4f07e73 x1874223835482240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787399498 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12020.197476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.198911] LustreError: 395775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.198921] LustreError: 395775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12020.228424] Lustre: 387025:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12021.017221] LustreError: 395756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399499 with bad export cookie 2117934563435038483 [12021.034592] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12021.366859] Lustre: server umount lustre-MDT0001 complete [12034.513469] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12037.439613] Key type lgssc unregistered [12037.864932] LNet: 400236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12037.876533] LNetError: 400236:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12037.888249] LNet: Removed LNI 192.168.204.131@tcp [12038.770848] Key type .llcrypt unregistered [12038.776075] Key type ._llcrypt unregistered [12065.649604] Key type ._llcrypt registered [12065.651428] Key type .llcrypt registered [12065.769918] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12084.852472] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12085.885793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12085.937102] alg: No test for adler32 (adler32-zlib) [12086.980821] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12087.190902] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12088.847200] Key type lgssc registered [12090.138873] Lustre: Echo OBD driver; http://www.lustre.org/ [12131.858135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12131.951928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12133.411950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12133.465740] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12133.626621] Lustre: lustre-MDT0000: new disk, initializing [12133.836618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12133.871181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12139.519407] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12151.229301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12151.381516] Lustre: 403397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12151.443088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12151.458234] Lustre: Skipped 1 previous similar message [12151.533903] Lustre: lustre-MDT0001: new disk, initializing [12151.669449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12151.711084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12151.733433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12156.993508] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12168.330943] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12174.418888] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12182.372170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.790444] Lustre: lustre-OST0000: new disk, initializing [12182.794244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12182.800796] Lustre: 405367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12182.893035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12189.763288] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12189.809264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12189.831599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12190.011130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12201.346252] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12207.212809] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12207.477786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12212.674508] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12212.923344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12215.779078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.781859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12215.782878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12215.801514] Lustre: Skipped 1 previous similar message [12215.837552] LustreError: Skipped 1 previous similar message [12220.900455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12220.916879] Lustre: Skipped 2 previous similar messages [12221.774661] Lustre: server umount lustre-OST0000 complete [12228.579679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12228.591651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12228.620674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12231.143064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12231.144153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12231.181542] Lustre: Skipped 1 previous similar message [12232.899687] Lustre: server umount lustre-MDT0000 complete [12237.532438] LustreError: 403388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399715 with bad export cookie 1136112301178525643 [12237.536093] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12237.547942] LustreError: 403388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12238.124314] Lustre: server umount lustre-MDT0001 complete [12248.441888] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12251.195554] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 07:55:27 (1787399727) [12260.118867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12260.699430] LustreError: 407827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12260.733314] LustreError: 407827:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12260.832173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12265.952268] LustreError: 407828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12266.228536] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12270.085960] LustreError: 407827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.172645] LustreError: 407828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12277.612071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12278.008968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12283.986645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12293.060562] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12299.554784] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12306.769639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12307.102490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12313.562419] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12321.245653] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12323.369388] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12335.538040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12341.218951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.223149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12341.224490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12341.247722] Lustre: Skipped 1 previous similar message [12347.366635] Lustre: server umount lustre-OST0000 complete [12351.456764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12351.468070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12351.469033] Lustre: Skipped 1 previous similar message [12351.491831] Lustre: Skipped 4 previous similar messages [12357.706558] Lustre: server umount lustre-MDT0000 complete [12361.671856] LustreError: 407807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399839 with bad export cookie 1136112301178526525 [12361.682574] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12361.690718] LustreError: 407807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12362.036903] Lustre: server umount lustre-MDT0001 complete [12373.504441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12376.858381] Key type lgssc unregistered [12377.277631] LNet: 411580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12377.288608] LNetError: 411580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12377.308791] LNet: Removed LNI 192.168.204.131@tcp [12378.428386] Key type .llcrypt unregistered [12378.432611] Key type ._llcrypt unregistered [12407.893525] Key type ._llcrypt registered [12407.896438] Key type .llcrypt registered [12408.039572] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12408.414557] Key type .llcrypt unregistered [12408.420554] Key type ._llcrypt unregistered [12431.292485] Key type ._llcrypt registered [12431.293990] Key type .llcrypt registered [12431.405322] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12432.739203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12432.803241] alg: No test for adler32 (adler32-zlib) [12433.956780] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12434.353614] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12436.177316] Key type lgssc registered [12437.738840] Lustre: Echo OBD driver; http://www.lustre.org/ [12446.902763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12446.921788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12448.508853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12453.799551] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12457.963913] LustreError: 413801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12457.975592] LustreError: 413801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12463.073077] LustreError: 413800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12463.152726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12463.432180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12468.705615] LustreError: 413797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12468.751067] LustreError: 413797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12470.655290] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12473.827125] LustreError: 413801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12473.835910] LustreError: 413801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12473.844242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12478.802661] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12478.958363] LustreError: 413796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12478.970851] LustreError: 413796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12487.356431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12487.827253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12493.400740] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12504.551372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12504.558067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12504.578680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12508.130240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12508.141362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12508.152148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12510.646839] Lustre: server umount lustre-OST0000 complete [12515.791159] Lustre: server umount lustre-MDT0000 complete [12518.369975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12518.383784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12518.406717] LustreError: 415778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12518.442667] LustreError: 415778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12520.693543] LustreError: 413781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399998 with bad export cookie 15115834628623169562 [12520.698874] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12520.722967] LustreError: 413781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12521.176979] Lustre: server umount lustre-MDT0001 complete [12534.156759] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12537.355945] Key type lgssc unregistered [12537.656598] LNet: 416949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12537.664537] LNetError: 416949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12537.703576] LNet: Removed LNI 192.168.204.131@tcp [12539.010334] Key type .llcrypt unregistered [12539.015187] Key type ._llcrypt unregistered [12565.990359] Key type ._llcrypt registered [12565.993151] Key type .llcrypt registered [12566.132531] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12567.388041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12567.451477] alg: No test for adler32 (adler32-zlib) [12568.732286] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12569.092318] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12570.847258] Key type lgssc registered [12572.096995] Lustre: Echo OBD driver; http://www.lustre.org/ [12582.323234] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12582.366377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12597.983500] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12603.103661] LustreError: 418340:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [12629.855456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12638.089391] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12647.635817] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12656.041398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12656.659426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12656.782158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12661.737240] LustreError: 418365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.755874] LustreError: 418365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12661.791662] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12664.820953] LustreError: 418364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.849203] LustreError: 418364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12669.924559] LustreError: 418365:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12669.949425] LustreError: 418365:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12671.879109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.387791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12677.956617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12687.424516] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12694.278683] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12707.825107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12707.838890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12707.852779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12709.802251] Lustre: server umount lustre-OST0000 complete [12716.528085] Lustre: server umount lustre-MDT0000 complete [12718.563791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12718.571901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12718.589523] LustreError: Skipped 1 previous similar message [12718.624090] Lustre: Skipped 1 previous similar message [12718.649578] LustreError: 419393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12718.671555] LustreError: 419393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12721.395640] LustreError: 418348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400199 with bad export cookie 12312464417277821240 [12721.404342] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12721.408100] LustreError: 418348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12721.931539] Lustre: server umount lustre-MDT0001 complete [12734.546687] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12737.491401] Key type lgssc unregistered [12737.861398] LNet: 421924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12737.868282] LNetError: 421924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12737.891210] LNet: Removed LNI 192.168.204.131@tcp [12738.870947] Key type .llcrypt unregistered [12738.876948] Key type ._llcrypt unregistered [12765.338291] Key type ._llcrypt registered [12765.342802] Key type .llcrypt registered [12765.451778] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12766.713879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12766.871763] alg: No test for adler32 (adler32-zlib) [12768.066800] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12768.405643] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12770.120094] Key type lgssc registered [12771.547629] Lustre: Echo OBD driver; http://www.lustre.org/ [12781.107483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12781.125325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12782.593884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12787.434976] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12792.804093] LustreError: 423343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12792.822473] LustreError: 423343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12797.921968] LustreError: 423342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12799.746440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12800.215401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12806.806984] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12816.486874] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12823.083086] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12830.297692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12830.726355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12838.405216] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12846.846633] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12852.276219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12856.289442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.303247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.336544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12857.317512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12857.322314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12857.335577] Lustre: Skipped 1 previous similar message [12857.343691] Lustre: Skipped 1 previous similar message [12859.040264] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.31@tcp (stopping) [12859.046222] Lustre: Skipped 1 previous similar message [12862.447427] LustreError: 423339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12862.488665] LustreError: 423339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12862.700108] Lustre: server umount lustre-MDT0000 complete [12865.256364] LustreError: 423325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.31@tcp arrived at 1787400343 with bad export cookie 5593234111260425540 [12865.270065] LustreError: 423325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12867.558427] LustreError: 423342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.567165] LustreError: 423323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400345 with bad export cookie 5593234111260424875 [12867.590575] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.597909] LustreError: 423342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12867.612992] LustreError: 423323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12870.626201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12870.640466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12870.655822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12872.677576] LustreError: 424071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.714714] LustreError: 424071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12873.881367] Lustre: server umount lustre-OST0000 complete [12881.836743] Lustre: server umount lustre-MDT0001 complete [12893.353197] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12896.302931] Key type lgssc unregistered [12896.594310] LNet: 427031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12896.604542] LNetError: 427031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12896.626456] LNet: Removed LNI 192.168.204.131@tcp [12897.530448] Key type .llcrypt unregistered [12897.533567] Key type ._llcrypt unregistered [12924.476850] Key type ._llcrypt registered [12924.481880] Key type .llcrypt registered [12924.650115] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12926.466128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12926.534569] alg: No test for adler32 (adler32-zlib) [12927.729583] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [12928.034476] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12929.793156] Key type lgssc registered [12931.652933] Lustre: Echo OBD driver; http://www.lustre.org/ [12941.569393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12941.596039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12942.974081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12947.695618] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12953.576938] LustreError: 428450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12953.629557] LustreError: 428450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12958.697318] LustreError: 428449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12958.783429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12959.529303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12965.205922] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12974.875407] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12981.364873] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12988.835815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12989.245163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12990.272437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12995.925423] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13003.750795] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13010.920134] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13010.932303] Lustre: Skipped 1 previous similar message [13010.932363] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13010.934224] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13011.631911] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [13011.642355] Lustre: Skipped 1 previous similar message [13015.281385] Lustre: server umount lustre-MDT0001 complete [13016.054662] LustreError: 428450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13016.091360] LustreError: 428450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13021.159120] LustreError: 428449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13021.191444] LustreError: 428449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13026.278991] LustreError: 429410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.281396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13026.311746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.329455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13028.487603] Lustre: server umount lustre-OST0000 complete [13033.815497] Lustre: server umount lustre-MDT0000 complete [13050.058375] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13053.551317] Key type lgssc unregistered [13053.905871] LNet: 432134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13053.911818] LNetError: 432134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13053.922150] LNet: Removed LNI 192.168.204.131@tcp [13054.791177] Key type .llcrypt unregistered [13054.794674] Key type ._llcrypt unregistered [13089.610675] Key type ._llcrypt registered [13089.615140] Key type .llcrypt registered [13089.797618] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13106.288447] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13107.351827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13107.459615] alg: No test for adler32 (adler32-zlib) [13108.513478] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [13108.710873] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [13110.375172] Key type lgssc registered [13111.717358] Lustre: Echo OBD driver; http://www.lustre.org/ [13153.641834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13170.862879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13170.895248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13172.131582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13172.168719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13172.249499] Lustre: lustre-MDT0000: new disk, initializing [13172.351668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13172.370573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13177.402390] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13196.701548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13197.161878] Lustre: lustre-OST0000: new disk, initializing [13197.177079] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13197.183701] Lustre: Skipped 1 previous similar message [13197.189482] Lustre: 437173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13197.257277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13204.638702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13204.661378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13204.709554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13205.232288] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13215.615692] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13232.786617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13233.025873] Lustre: 436155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13233.109262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13233.208228] Lustre: lustre-MDT0001: new disk, initializing [13233.333631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13233.374441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13233.382332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13238.627770] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13248.699369] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13249.034234] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13253.947267] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13254.166883] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13263.330126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13263.342245] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13263.343158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13263.357569] Lustre: Skipped 1 previous similar message [13265.453396] Lustre: server umount lustre-MDT0001 complete [13273.571059] LustreError: 436161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.586766] LustreError: 436161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13278.697907] LustreError: 438888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13278.716172] LustreError: 438888:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13283.809696] LustreError: 436165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13283.813372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13283.843285] Lustre: Skipped 1 previous similar message [13283.855535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13283.861515] Lustre: Skipped 1 previous similar message [13288.929301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13288.929833] LustreError: 436166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13289.099583] Lustre: server umount lustre-OST0000 complete [13298.893583] Lustre: server umount lustre-MDT0000 complete [13311.433525] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13314.297837] Key type lgssc unregistered [13314.599751] LNet: 440921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13314.617220] LNetError: 440921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13314.630308] LNet: Removed LNI 192.168.204.131@tcp [13315.462098] Key type .llcrypt unregistered [13315.467151] Key type ._llcrypt unregistered [13353.452524] Key type ._llcrypt registered [13353.455922] Key type .llcrypt registered [13353.575534] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13353.946379] Key type .llcrypt unregistered [13353.951638] Key type ._llcrypt unregistered [13371.478609] Key type ._llcrypt registered [13371.483615] Key type .llcrypt registered [13371.629223] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13372.699937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13372.798787] alg: No test for adler32 (adler32-zlib) [13373.913762] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [13374.119985] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [13375.807496] Key type lgssc registered [13377.127650] Lustre: Echo OBD driver; http://www.lustre.org/ [13404.170932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13404.219016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13405.592241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13405.617626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13405.704508] Lustre: lustre-MDT0000: new disk, initializing [13405.760249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13405.778898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13410.009055] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13419.126974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13419.223631] Lustre: 444298:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13419.310199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13419.320063] Lustre: Skipped 1 previous similar message [13419.404355] Lustre: lustre-MDT0001: new disk, initializing [13419.489348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13419.522036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13419.536258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13423.509269] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13432.775336] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13438.742883] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13445.421701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13445.665461] Lustre: lustre-OST0000: new disk, initializing [13445.672627] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13445.691866] Lustre: 446268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13445.786091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13451.568383] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13454.465064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13454.480484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13454.650459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13460.925355] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13470.185473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.191169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13470.206322] Lustre: Skipped 2 previous similar messages [13470.222024] Lustre: Skipped 2 previous similar messages [13475.246549] Lustre: server umount lustre-MDT0000 complete [13478.832456] LustreError: 444292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400956 with bad export cookie 15783169687340804500 [13478.851188] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13478.856982] LustreError: 444292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13479.209784] Lustre: server umount lustre-MDT0001 complete [13493.037893] Lustre: server umount lustre-OST0000 complete [13503.376887] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 08:16:20 (1787400980) [13510.358052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13510.808613] LustreError: 448295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13510.825938] LustreError: 448295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13510.900305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13515.346379] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13516.258123] LustreError: 448296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13520.356382] LustreError: 448295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13523.945216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13524.269665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13528.687483] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13536.540883] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13542.083397] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13562.027988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13562.245130] Lustre: Found index 0 for lustre-OST0000, updating log [13562.441738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13568.231215] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13574.215354] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13581.878830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13589.984224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13589.990357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13590.001115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13591.072977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13591.092161] Lustre: Skipped 2 previous similar messages [13595.453696] Lustre: server umount lustre-OST0000 complete [13611.489419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13611.509433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13611.540575] Lustre: Skipped 1 previous similar message [13611.554404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13617.633471] LustreError: 448295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13617.678702] Lustre: server umount lustre-MDT0000 complete [13621.600332] LustreError: 448277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401099 with bad export cookie 15783169687340805886 [13621.601288] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13621.624411] LustreError: 448277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13621.907155] Lustre: server umount lustre-MDT0001 complete [13626.017609] LNet: 1 local NIs in recovery (showing 1): 192.168.204.131@tcp [13633.038712] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 08:18:29 (1787401109) [14249.504025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14249.892783] LustreError: 454729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14249.913911] LustreError: 454729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [14249.967568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14253.757586] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14255.073729] LustreError: 454730:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14259.172017] LustreError: 454729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14261.580557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14261.901892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14265.823266] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14272.688780] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14277.589604] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14297.514474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14297.844728] Lustre: Found index 0 for lustre-OST0000, updating log [14298.098249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14303.399455] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14303.905122] LNet: 1 local NIs in recovery (showing 1): 192.168.204.131@tcp [14311.693820] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14314.460164] Lustre: server umount lustre-OST0000 complete [14318.048102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14318.061724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14318.074112] Lustre: Skipped 1 previous similar message [14318.078112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14318.082603] Lustre: Skipped 2 previous similar messages [14319.138775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14319.156536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14320.230042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14320.242770] Lustre: Skipped 2 previous similar messages [14322.273969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14322.277779] Lustre: Skipped 3 previous similar messages [14326.371053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14326.375406] Lustre: Skipped 7 previous similar messages [14332.385169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14332.514890] LustreError: 455458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14332.527338] LustreError: 455458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14332.554602] Lustre: server umount lustre-MDT0000 complete [14336.358338] LustreError: 454710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401814 with bad export cookie 15783169687340806978 [14336.361962] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14336.368316] LustreError: 454710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14336.812552] Lustre: server umount lustre-MDT0001 complete [14363.300119] LNet: 1 local NIs in recovery (showing 1): 192.168.204.131@tcp [14366.714237] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [14375.688205] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14411.986901] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 08:31:28 (1787401888) [14427.782166] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14430.211415] Key type lgssc unregistered [14430.501363] LNet: 461335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14430.507632] LNetError: 461335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14558.502934] LNet: Removed LNI 192.168.204.131@tcp [14559.525812] Key type .llcrypt unregistered [14559.528941] Key type ._llcrypt unregistered [14577.115587] Key type ._llcrypt registered [14577.118561] Key type .llcrypt registered [14577.269853] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14578.406109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14578.441795] alg: No test for adler32 (adler32-zlib) [14579.516179] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [14579.848720] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [14581.639504] Key type lgssc registered [14582.879800] Lustre: Echo OBD driver; http://www.lustre.org/ [14589.465507] LNet: Added LNI 192.168.204.131@tcp100 [8/256/0/180] [14590.394528] LNet: Added LNI 192.168.204.131@tcp101 [8/256/0/180] [14591.472714] LNet: Added LNI 192.168.204.131@tcp102 [8/256/0/180] [14592.408110] LNet: Added LNI 192.168.204.131@tcp103 [8/256/0/180] [14593.427415] LNet: Added LNI 192.168.204.131@tcp104 [8/256/0/180] [14594.492655] LNet: Added LNI 192.168.204.131@tcp105 [8/256/0/180] [14595.768539] LNet: Added LNI 192.168.204.131@tcp106 [8/256/0/180] [14596.817250] LNet: Added LNI 192.168.204.131@tcp107 [8/256/0/180] [14597.907891] LNet: Added LNI 192.168.204.131@tcp108 [8/256/0/180] [14599.082578] LNet: Added LNI 192.168.204.131@tcp109 [8/256/0/180] [14600.087000] LNet: Added LNI 192.168.204.131@tcp110 [8/256/0/180] [14601.017689] LNet: Added LNI 192.168.204.131@tcp111 [8/256/0/180] [14601.856899] LNet: Added LNI 192.168.204.131@tcp112 [8/256/0/180] [14602.766341] LNet: Added LNI 192.168.204.131@tcp113 [8/256/0/180] [14603.719740] LNet: Added LNI 192.168.204.131@tcp114 [8/256/0/180] [14604.710255] LNet: Added LNI 192.168.204.131@tcp115 [8/256/0/180] [14605.640943] LNet: Added LNI 192.168.204.131@tcp116 [8/256/0/180] [14606.840643] LNet: Added LNI 192.168.204.131@tcp117 [8/256/0/180] [14608.143513] LNet: Added LNI 192.168.204.131@tcp118 [8/256/0/180] [14609.260701] LNet: Added LNI 192.168.204.131@tcp119 [8/256/0/180] [14610.596884] LNet: Added LNI 192.168.204.131@tcp120 [8/256/0/180] [14611.936417] LNet: Added LNI 192.168.204.131@tcp121 [8/256/0/180] [14612.924680] LNet: Added LNI 192.168.204.131@tcp122 [8/256/0/180] [14613.989995] LNet: Added LNI 192.168.204.131@tcp123 [8/256/0/180] [14615.150140] LNet: Added LNI 192.168.204.131@tcp124 [8/256/0/180] [14616.234940] LNet: Added LNI 192.168.204.131@tcp125 [8/256/0/180] [14617.343475] LNet: Added LNI 192.168.204.131@tcp126 [8/256/0/180] [14618.381947] LNet: Added LNI 192.168.204.131@tcp127 [8/256/0/180] [14619.522062] LNet: Added LNI 192.168.204.131@tcp128 [8/256/0/180] [14620.613981] LNet: Added LNI 192.168.204.131@tcp129 [8/256/0/180] [14621.950447] LNet: Added LNI 192.168.204.131@tcp130 [8/256/0/180] [14623.167238] LNet: Added LNI 192.168.204.131@tcp131 [8/256/0/180] [14624.540689] LNet: Added LNI 192.168.204.131@tcp132 [8/256/0/180] [14625.491871] LNet: Added LNI 192.168.204.131@tcp133 [8/256/0/180] [14626.357284] LNet: Added LNI 192.168.204.131@tcp134 [8/256/0/180] [14627.331606] LNet: Added LNI 192.168.204.131@tcp135 [8/256/0/180] [14634.304370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14634.334358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14635.490142] Lustre: 464371:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.131@tcp: can't add NID 192.168.204.131@tcp130: rc = -75 [14635.569660] LNetError: 464392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14635.581467] LNetError: Protocol error connecting to 192.168.204.131@tcp116 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14635.706756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14635.752349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14635.823294] Lustre: lustre-MDT0000: new disk, initializing [14635.918581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14635.938062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14636.191298] LNet: 4 local NIs in recovery (showing 4): 192.168.204.131@tcp126, 192.168.204.131@tcp125, 192.168.204.131@tcp127, 192.168.204.131@tcp129 [14636.192204] LNetError: 461829:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14636.226487] LNetError: 461829:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [14636.247285] LNetError: Protocol error connecting to 192.168.204.131@tcp109 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14636.262464] LNetError: Skipped 8 previous similar messages [14640.959811] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14654.022650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14654.288689] Lustre: 464389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14654.474105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14654.480529] Lustre: Skipped 1 previous similar message [14654.609335] Lustre: lustre-MDT0001: new disk, initializing [14654.664784] Lustre: 465190:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.131@tcp: can't add NID 192.168.204.131@tcp130: rc = -75 [14654.677415] Lustre: 465190:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14654.849490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14654.944102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14654.954620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14660.835609] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14663.302854] Lustre: 465540:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.131@tcp: can't add NID 192.168.204.131@tcp130: rc = -75 [14663.322671] Lustre: 465540:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14665.890629] LNetError: 461830:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14665.916482] LNetError: Protocol error connecting to 192.168.204.131@tcp122 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14668.967379] LNetError: 464405:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14668.975646] LNetError: 464405:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14668.982657] LNetError: Protocol error connecting to 192.168.204.131@tcp120 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14668.993379] LNetError: Skipped 1 previous similar message [14673.431835] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14674.080969] LNetError: 464392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14674.093365] LNetError: 464392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14674.100436] LNetError: Protocol error connecting to 192.168.204.131@tcp116 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14674.106018] LNetError: Skipped 1 previous similar message [14680.415122] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14683.297881] LNetError: 464400:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14683.308406] LNetError: 464400:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14683.325892] LNetError: Protocol error connecting to 192.168.204.131@tcp128 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14683.342154] LNetError: Skipped 4 previous similar messages [14690.648414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14690.949448] Lustre: lustre-OST0000: new disk, initializing [14690.953360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14690.960529] Lustre: 466363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14691.045706] Lustre: 466368:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.131@tcp: can't add NID 192.168.204.131@tcp130: rc = -75 [14691.052972] Lustre: 466368:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14691.098849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14697.730253] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14700.048963] Lustre: 466821:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.131@tcp: can't add NID 192.168.204.131@tcp130: rc = -75 [14700.062208] Lustre: 466821:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14700.119063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14700.129986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14700.208384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14700.705457] LNetError: 461831:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14700.720227] LNetError: 461831:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14700.729195] LNetError: Protocol error connecting to 192.168.204.131@tcp124 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14700.741415] LNetError: Skipped 6 previous similar messages [14707.018167] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14715.878238] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14715.879095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14715.924686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14715.940811] Lustre: Skipped 1 previous similar message [14720.752228] Lustre: server umount lustre-OST0000 complete [14726.120643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14726.122845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14726.135703] Lustre: Skipped 2 previous similar messages [14726.174222] Lustre: Skipped 1 previous similar message [14731.283113] Lustre: server umount lustre-MDT0000 complete [14734.511082] LNetError: 464392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.131 [14734.525254] LNetError: 464392:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [14734.537283] LNetError: Protocol error connecting to 192.168.204.131@tcp128 at host 192.168.204.131:988: is it running a compatible version of Lustre? [14734.549902] LNetError: Skipped 7 previous similar messages [14735.485929] LustreError: 464380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402213 with bad export cookie 18086136162868661850 [14735.491453] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14735.499582] LustreError: 464380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14735.967309] Lustre: server umount lustre-MDT0001 complete [14748.361927] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14751.107879] Key type lgssc unregistered [14751.525913] LNet: 468278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14751.543152] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14751.549098] LNet: Removed LNI 192.168.204.131@tcp [14752.607443] LNet: Removed LNI 192.168.204.131@tcp100 [14752.610740] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14752.627781] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14752.635235] LNet: Removed LNI 192.168.204.131@tcp101 [14753.695784] LNet: Removed LNI 192.168.204.131@tcp102 [14753.715435] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14753.725246] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14753.731390] LNet: Removed LNI 192.168.204.131@tcp103 [14754.783788] LNet: Removed LNI 192.168.204.131@tcp104 [14755.807529] LNet: Removed LNI 192.168.204.131@tcp105 [14755.809641] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14755.817571] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14755.825425] LNet: Removed LNI 192.168.204.131@tcp106 [14756.831157] LNet: Removed LNI 192.168.204.131@tcp107 [14757.855189] LNet: Removed LNI 192.168.204.131@tcp108 [14758.879184] LNet: Removed LNI 192.168.204.131@tcp109 [14759.903155] LNet: Removed LNI 192.168.204.131@tcp110 [14759.916813] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14759.930932] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14759.947574] LNet: Removed LNI 192.168.204.131@tcp111 [14760.991340] LNet: Removed LNI 192.168.204.131@tcp112 [14762.015168] LNet: Removed LNI 192.168.204.131@tcp113 [14763.040982] LNet: Removed LNI 192.168.204.131@tcp114 [14764.063135] LNet: Removed LNI 192.168.204.131@tcp115 [14765.088455] LNet: Removed LNI 192.168.204.131@tcp116 [14766.111164] LNet: Removed LNI 192.168.204.131@tcp117 [14767.135171] LNet: Removed LNI 192.168.204.131@tcp118 [14768.159651] LNet: Removed LNI 192.168.204.131@tcp119 [14768.161655] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14768.168869] LNetError: 468278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14768.176486] LNet: Removed LNI 192.168.204.131@tcp120 [14769.247229] LNet: Removed LNI 192.168.204.131@tcp121 [14770.271250] LNet: Removed LNI 192.168.204.131@tcp122 [14771.295570] LNet: Removed LNI 192.168.204.131@tcp123 [14772.319321] LNet: Removed LNI 192.168.204.131@tcp124 [14773.343608] LNet: Removed LNI 192.168.204.131@tcp125 [14774.367198] LNet: Removed LNI 192.168.204.131@tcp126 [14775.391632] LNet: Removed LNI 192.168.204.131@tcp127 [14776.415238] LNet: Removed LNI 192.168.204.131@tcp128 [14777.439178] LNet: Removed LNI 192.168.204.131@tcp129 [14777.447155] LNet: Removed LNI 192.168.204.131@tcp130 [14777.453989] LNet: Removed LNI 192.168.204.131@tcp131 [14777.458126] LNet: Removed LNI 192.168.204.131@tcp132 [14777.465126] LNet: Removed LNI 192.168.204.131@tcp133 [14777.471483] LNet: Removed LNI 192.168.204.131@tcp134 [14777.487882] LNet: Removed LNI 192.168.204.131@tcp135 [14778.486166] Key type .llcrypt unregistered [14778.495223] Key type ._llcrypt unregistered [14804.616309] Key type ._llcrypt registered [14804.619585] Key type .llcrypt registered [14804.817075] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [14821.309622] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14822.830769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14822.855525] alg: No test for adler32 (adler32-zlib) [14823.997870] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [14824.303504] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [14825.991227] Key type lgssc registered [14827.606833] Lustre: Echo OBD driver; http://www.lustre.org/ [14866.280750] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 08:39:02 (1787402342) [14885.340190] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14888.297926] Key type lgssc unregistered [14888.629556] LNet: 472370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14888.651157] LNetError: 472370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14888.664940] LNet: Removed LNI 192.168.204.131@tcp [14889.502642] Key type .llcrypt unregistered [14889.504829] Key type ._llcrypt unregistered [14907.964074] Key type ._llcrypt registered [14907.966787] Key type .llcrypt registered [14908.137852] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14909.359178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14909.484922] alg: No test for adler32 (adler32-zlib) [14910.582874] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [14910.893861] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [14912.591190] Key type lgssc registered [14913.793566] Lustre: Echo OBD driver; http://www.lustre.org/ [14924.110788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14924.165124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14925.434374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14925.477492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14925.575383] Lustre: lustre-MDT0000: new disk, initializing [14925.675051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14925.692648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14930.139445] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14940.000381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14940.095196] Lustre: 473584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14940.149481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14940.154546] Lustre: Skipped 1 previous similar message [14940.218603] Lustre: lustre-MDT0001: new disk, initializing [14940.309671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14940.358371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14940.375089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14945.740897] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14957.464652] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14964.212608] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14972.652813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14972.969621] Lustre: lustre-OST0000: new disk, initializing [14972.977330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14972.994915] Lustre: 475553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14973.116718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14978.648740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14978.657209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14978.796030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14980.085519] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14990.407498] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14994.742896] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing lnet_if_list [15000.253921] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15000.290885] LNet: Added LNI 192.168.204.131@tcp10 [8/256/0/180] [15006.352677] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15006.378447] LNet: Added LNI 192.168.204.131@tcp11 [8/256/0/180] [15010.406078] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15010.438123] LNet: Added LNI 192.168.204.131@tcp12 [8/256/0/180] [15014.564617] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15014.588421] LNet: Added LNI 192.168.204.131@tcp13 [8/256/0/180] [15019.726215] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15019.749632] LNet: Added LNI 192.168.204.131@tcp14 [8/256/0/180] [15025.010160] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15025.041284] LNet: Added LNI 192.168.204.131@tcp15 [8/256/0/180] [15030.161395] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15030.188683] LNet: Added LNI 192.168.204.131@tcp16 [8/256/0/180] [15035.871943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15035.919797] LNet: Added LNI 192.168.204.131@tcp17 [8/256/0/180] [15042.212187] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15042.247560] LNet: Added LNI 192.168.204.131@tcp18 [8/256/0/180] [15048.430734] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15048.461351] LNet: Added LNI 192.168.204.131@tcp19 [8/256/0/180] [15054.840115] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15054.903510] LNet: Added LNI 192.168.204.131@tcp20 [8/256/0/180] [15061.256261] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15061.294676] LNet: Added LNI 192.168.204.131@tcp21 [8/256/0/180] [15068.031792] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15068.084156] LNet: Added LNI 192.168.204.131@tcp22 [8/256/0/180] [15074.712369] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15074.744656] LNet: Added LNI 192.168.204.131@tcp23 [8/256/0/180] [15081.226786] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15081.268086] LNet: Added LNI 192.168.204.131@tcp24 [8/256/0/180] [15087.555846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15087.592717] LNet: Added LNI 192.168.204.131@tcp25 [8/256/0/180] [15094.271952] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15094.301586] LNet: Added LNI 192.168.204.131@tcp26 [8/256/0/180] [15101.114638] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15101.153793] LNet: Added LNI 192.168.204.131@tcp27 [8/256/0/180] [15108.553321] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15108.610292] LNet: Added LNI 192.168.204.131@tcp28 [8/256/0/180] [15116.146301] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15116.172876] LNet: Added LNI 192.168.204.131@tcp29 [8/256/0/180] [15123.820533] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15123.859990] LNet: Added LNI 192.168.204.131@tcp30 [8/256/0/180] [15131.357299] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15131.391843] LNet: Added LNI 192.168.204.131@tcp31 [8/256/0/180] [15138.313102] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15138.364605] LNet: Added LNI 192.168.204.131@tcp32 [8/256/0/180] [15145.568531] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15145.619364] LNet: Added LNI 192.168.204.131@tcp33 [8/256/0/180] [15152.469174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15152.495104] LNet: Added LNI 192.168.204.131@tcp34 [8/256/0/180] [15159.756174] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15159.806644] LNet: Added LNI 192.168.204.131@tcp35 [8/256/0/180] [15168.039432] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15168.081891] LNet: Added LNI 192.168.204.131@tcp36 [8/256/0/180] [15175.747775] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15175.787751] LNet: Added LNI 192.168.204.131@tcp37 [8/256/0/180] [15183.523137] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15183.542689] LNet: Added LNI 192.168.204.131@tcp38 [8/256/0/180] [15191.719625] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15191.757664] LNet: Added LNI 192.168.204.131@tcp39 [8/256/0/180] [15201.070220] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15201.109976] LNet: Added LNI 192.168.204.131@tcp40 [8/256/0/180] [15209.739382] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15209.773905] LNet: Added LNI 192.168.204.131@tcp41 [8/256/0/180] [15218.121694] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15218.147494] LNet: Added LNI 192.168.204.131@tcp42 [8/256/0/180] [15226.528335] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15226.589645] LNet: Added LNI 192.168.204.131@tcp43 [8/256/0/180] [15234.769371] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15234.804196] LNet: Added LNI 192.168.204.131@tcp44 [8/256/0/180] [15243.289879] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15243.360421] LNet: Added LNI 192.168.204.131@tcp45 [8/256/0/180] [15252.528846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15252.561052] LNet: Added LNI 192.168.204.131@tcp46 [8/256/0/180] [15262.094293] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15262.132222] LNet: Added LNI 192.168.204.131@tcp47 [8/256/0/180] [15271.744773] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15271.785369] LNet: Added LNI 192.168.204.131@tcp48 [8/256/0/180] [15281.009144] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15281.061960] LNet: Added LNI 192.168.204.131@tcp49 [8/256/0/180] [15290.706769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15290.731744] LNet: Added LNI 192.168.204.131@tcp50 [8/256/0/180] [15314.401172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15314.421744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15314.436645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15317.526642] Lustre: server umount lustre-MDT0000 complete [15321.421651] LustreError: 473578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402799 with bad export cookie 3952754339883269709 [15321.424475] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15321.432466] LustreError: 473578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15321.853543] Lustre: server umount lustre-MDT0001 complete [15341.540275] Lustre: 472856:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402803/real 1787402803] req@0000000031feb58b x1874227246583680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402819 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15341.566209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15341.586245] Lustre: Skipped 2 previous similar messages [15342.566087] Lustre: 472859:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402804/real 1787402804] req@000000008052dc91 x1874227246583808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15343.867103] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15346.660457] Lustre: 472858:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402808/real 1787402808] req@00000000aa30ad4a x1874227246583936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402824 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15352.799234] Lustre: 472857:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402814/real 1787402814] req@000000005da33cad x1874227246584320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15352.824216] Lustre: 472857:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15353.552438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15357.919165] Lustre: 472859:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787402819/real 1787402819] req@000000000e6b31d3 x1874227246584448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787402835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15366.553541] LustreError: 475546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15366.574037] LustreError: 475546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [15366.636568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15370.720811] LustreError: 475547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15370.729982] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15371.748528] LustreError: 475546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15372.111401] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15376.875448] LustreError: 475547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15381.466670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15382.142710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15386.283772] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15387.353402] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15396.495598] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15403.114304] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15427.559901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15427.561303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15427.566433] Lustre: Skipped 1 previous similar message [15427.586382] Lustre: Skipped 3 previous similar messages [15429.537695] Lustre: server umount lustre-OST0000 complete [15433.185070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15433.190463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15433.199837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15437.792683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15437.795644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15437.819163] Lustre: Skipped 1 previous similar message [15439.350345] Lustre: server umount lustre-MDT0000 complete [15442.914735] LustreError: 499803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15442.939548] LustreError: 499803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [15443.358319] LustreError: 473576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402921 with bad export cookie 3952754339883272376 [15443.365252] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15443.377360] LustreError: 473576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15443.817269] Lustre: server umount lustre-MDT0001 complete [15457.567969] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15461.009933] Key type lgssc unregistered [15461.592700] LNet: 502599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15461.604254] LNetError: 502599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15461.615203] LNet: Removed LNI 192.168.204.131@tcp [15461.625204] LNet: Removed LNI 192.168.204.131@tcp10 [15461.635333] LNet: Removed LNI 192.168.204.131@tcp11 [15461.645481] LNet: Removed LNI 192.168.204.131@tcp12 [15461.653287] LNet: Removed LNI 192.168.204.131@tcp13 [15461.657052] LNet: Removed LNI 192.168.204.131@tcp14 [15461.660916] LNet: Removed LNI 192.168.204.131@tcp15 [15461.664468] LNet: Removed LNI 192.168.204.131@tcp16 [15461.668949] LNet: Removed LNI 192.168.204.131@tcp17 [15461.672993] LNet: Removed LNI 192.168.204.131@tcp18 [15461.677489] LNet: Removed LNI 192.168.204.131@tcp19 [15461.681527] LNet: Removed LNI 192.168.204.131@tcp20 [15461.683802] LNet: Removed LNI 192.168.204.131@tcp21 [15461.702343] LNet: Removed LNI 192.168.204.131@tcp22 [15461.705900] LNet: Removed LNI 192.168.204.131@tcp23 [15461.709971] LNet: Removed LNI 192.168.204.131@tcp24 [15461.713388] LNet: Removed LNI 192.168.204.131@tcp25 [15461.725057] LNet: Removed LNI 192.168.204.131@tcp26 [15461.733765] LNet: Removed LNI 192.168.204.131@tcp27 [15461.737386] LNet: Removed LNI 192.168.204.131@tcp28 [15461.740871] LNet: Removed LNI 192.168.204.131@tcp29 [15461.745762] LNet: Removed LNI 192.168.204.131@tcp30 [15461.752544] LNet: Removed LNI 192.168.204.131@tcp31 [15461.760503] LNet: Removed LNI 192.168.204.131@tcp32 [15461.763925] LNet: Removed LNI 192.168.204.131@tcp33 [15461.771582] LNet: Removed LNI 192.168.204.131@tcp34 [15461.778415] LNet: Removed LNI 192.168.204.131@tcp35 [15461.786846] LNet: Removed LNI 192.168.204.131@tcp36 [15461.790443] LNet: Removed LNI 192.168.204.131@tcp37 [15461.792872] LNet: Removed LNI 192.168.204.131@tcp38 [15461.802342] LNet: Removed LNI 192.168.204.131@tcp39 [15461.808273] LNet: Removed LNI 192.168.204.131@tcp40 [15461.818805] LNet: Removed LNI 192.168.204.131@tcp41 [15461.825506] LNet: Removed LNI 192.168.204.131@tcp42 [15461.838223] LNet: Removed LNI 192.168.204.131@tcp43 [15461.861669] LNet: Removed LNI 192.168.204.131@tcp44 [15461.879158] LNet: Removed LNI 192.168.204.131@tcp45 [15461.893288] LNet: Removed LNI 192.168.204.131@tcp46 [15461.895481] LNet: Removed LNI 192.168.204.131@tcp47 [15461.900323] LNet: Removed LNI 192.168.204.131@tcp48 [15461.915799] LNet: Removed LNI 192.168.204.131@tcp49 [15461.933892] LNet: Removed LNI 192.168.204.131@tcp50 [15463.148571] Key type .llcrypt unregistered [15463.152764] Key type ._llcrypt unregistered [15486.795376] Key type ._llcrypt registered [15486.797670] Key type .llcrypt registered [15486.941556] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [15504.835579] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15505.728288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15505.778294] alg: No test for adler32 (adler32-zlib) [15507.072861] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [15507.328480] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15509.024596] Key type lgssc registered [15510.142678] Lustre: Echo OBD driver; http://www.lustre.org/ [15550.119651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15550.150377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15551.458625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15551.481527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15551.623829] Lustre: lustre-MDT0000: new disk, initializing [15551.777977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15551.797339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15557.289736] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15569.586859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15569.780914] Lustre: 505798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15569.849200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15569.853354] Lustre: Skipped 1 previous similar message [15569.970832] Lustre: lustre-MDT0001: new disk, initializing [15570.111633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15570.191364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15570.207381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15575.813904] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15586.011923] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15592.790136] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15601.289730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15601.606871] Lustre: *** cfs_fail_loc=90f, val=0*** [15601.715871] Lustre: lustre-OST0000: new disk, initializing [15601.721519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15601.733289] Lustre: 507816:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15601.845933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15608.127352] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15610.002727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15610.025723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15610.098550] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15618.035913] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15625.699468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15625.700479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15625.701476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15625.714383] Lustre: Skipped 1 previous similar message [15625.725456] LustreError: Skipped 1 previous similar message [15630.830282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15630.838945] Lustre: Skipped 2 previous similar messages [15631.336434] Lustre: server umount lustre-OST0000 complete [15635.945517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15635.947535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15635.962496] Lustre: Skipped 1 previous similar message [15635.979952] Lustre: Skipped 1 previous similar message [15641.057745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15641.064262] Lustre: Skipped 1 previous similar message [15641.785756] Lustre: server umount lustre-MDT0000 complete [15645.288784] LustreError: 505791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403123 with bad export cookie 3624251991671186894 [15645.298600] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15645.304671] LustreError: 505791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15645.692937] Lustre: server umount lustre-MDT0001 complete [15657.807946] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15661.018277] Key type lgssc unregistered [15661.380840] LNet: 509721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15661.393346] LNetError: 509721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15661.432676] LNet: Removed LNI 192.168.204.131@tcp [15662.483794] Key type .llcrypt unregistered [15662.486166] Key type ._llcrypt unregistered [15682.673547] Key type ._llcrypt registered [15682.678369] Key type .llcrypt registered [15682.852384] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15684.296496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15684.473834] alg: No test for adler32 (adler32-zlib) [15685.695414] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [15686.129466] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15687.871934] Key type lgssc registered [15689.584660] Lustre: Echo OBD driver; http://www.lustre.org/ [15699.824591] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15699.850251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15701.191560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15706.247963] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15711.753116] LustreError: 511013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15711.795736] LustreError: 511013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [15716.554265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15716.836980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15717.114269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15722.601488] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15731.946898] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15739.834627] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15748.051834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15748.393555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15756.196219] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15764.633397] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15788.204953] loop0: detected capacity change from 0 to 204800000 [15788.408759] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15788.424319] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15788.432672] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15796.119414] loop0: detected capacity change from 0 to 204800000 [15796.383846] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15796.390773] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15796.401343] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15802.944103] loop0: detected capacity change from 0 to 204800000 [15809.696975] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15809.842485] Lustre: fs15246-OST0000: new disk, initializing [15809.846141] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15809.850831] Lustre: 514631:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15809.928821] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15817.980653] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15830.397963] loop1: detected capacity change from 0 to 204800000 [15837.268757] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15837.463515] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15837.539498] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15837.655716] Lustre: fs15246-MDT0000: new disk, initializing [15837.896348] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15837.923408] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15838.077743] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15839.142694] Lustre: 515755:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000163b12c x1874228060149120/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787403333 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15843.139818] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [15848.246242] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15848.276106] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15848.342420] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15870.959449] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15870.994997] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15876.072768] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15876.473037] Lustre: server umount fs15246-MDT0000 complete [15881.191039] LustreError: 513311:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15886.323501] LustreError: 511007:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15888.156106] Lustre: server umount fs15246-OST0000 complete [15905.251593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15905.260186] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15905.271329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15906.280899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15906.294322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15911.915273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15911.923926] Lustre: Skipped 3 previous similar messages [15917.025096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15917.210718] Lustre: server umount lustre-OST0000 complete [15922.146081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15922.160869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15922.168732] Lustre: Skipped 3 previous similar messages [15928.395592] Lustre: server umount lustre-MDT0000 complete [15932.229967] LustreError: 510994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403410 with bad export cookie 16606632220717439790 [15932.234604] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15932.241052] LustreError: 510994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15932.705901] Lustre: server umount lustre-MDT0001 complete [15943.480772] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [15946.507325] Key type lgssc unregistered [15946.910810] LNet: 518855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15946.924815] LNetError: 518855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15947.944499] LNet: Removed LNI 192.168.204.131@tcp [15948.763862] Key type .llcrypt unregistered [15948.769146] Key type ._llcrypt unregistered [15968.668543] Key type ._llcrypt registered [15968.671230] Key type .llcrypt registered [15968.817379] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [15969.908110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15969.949906] alg: No test for adler32 (adler32-zlib) [15971.008918] Lustre: Lustre: Build Version: 2.17.54_111_ga5eb910 [15971.242162] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [15972.954727] Key type lgssc registered [15974.183958] Lustre: Echo OBD driver; http://www.lustre.org/ [16017.553794] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [16029.419284] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [16074.032512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16074.046408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16075.213571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16075.262487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16075.335529] Lustre: lustre-MDT0000: new disk, initializing [16075.404490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16075.426841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16080.570822] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16092.618900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16092.817285] Lustre: 523323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16092.930375] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16092.936127] Lustre: Skipped 1 previous similar message [16093.126788] Lustre: lustre-MDT0001: new disk, initializing [16093.286934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16093.334451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16093.348115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16097.922993] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16107.333273] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16113.433910] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16120.491754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16120.769509] Lustre: lustre-OST0000: new disk, initializing [16120.775833] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16120.782400] Lustre: 525292:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16120.893706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16127.038816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16127.049294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16127.192056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16127.813662] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [16137.036817] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16142.275667] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16142.490404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16147.631251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16147.884660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16153.085428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16153.085637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16153.096221] LustreError: Skipped 1 previous similar message [16153.100577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16153.112954] Lustre: Skipped 1 previous similar message [16153.132234] Lustre: Skipped 1 previous similar message [16158.185309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16163.304301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16163.312857] Lustre: Skipped 2 previous similar messages [16164.320195] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16164.486495] Lustre: server umount lustre-OST0000 complete [16169.952275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16169.958955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16169.975375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16173.544835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16175.002603] Lustre: server umount lustre-MDT0000 complete [16178.896883] LustreError: 525291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403656 with bad export cookie 16821372680160803559 [16178.899303] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16178.906553] LustreError: 525291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16179.346681] Lustre: server umount lustre-MDT0001 complete [16208.729734] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [16220.000668] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [16255.678921] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15981 sec ========= 09:02:11 (1787403731) [16258.316756] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:02:14 (1787403734) === [16263.265918] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:02:19 (1787403739) === [16308.187761] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [16311.972355] Key type lgssc unregistered [16312.351768] LNet: 531952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16312.375661] LNetError: 531952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16312.395254] LNet: Removed LNI 192.168.204.131@tcp [16313.505820] Key type .llcrypt unregistered [16313.508855] Key type ._llcrypt unregistered