[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508371334 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003353] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008361] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010014] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011016] pid_max: default: 32768 minimum: 301 [ 0.013133] LSM: Security Framework initializing [ 0.014056] Yama: becoming mindful. [ 0.015037] SELinux: Initializing. [ 0.016063] *** VALIDATE selinux *** [ 0.024548] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030123] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034130] *** VALIDATE tmpfs *** [ 0.035480] *** VALIDATE proc *** [ 0.036271] *** VALIDATE cgroup *** [ 0.037012] *** VALIDATE cgroup2 *** [ 0.038284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042037] Spectre V2 : User space: Vulnerable [ 0.043011] Speculative Store Bypass: Vulnerable [ 0.046398] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.048188] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050027] ... version: 2 [ 0.051021] ... bit width: 48 [ 0.052013] ... generic registers: 4 [ 0.053014] ... value mask: 0000ffffffffffff [ 0.054017] ... max period: 00007fffffffffff [ 0.055017] ... fixed-purpose events: 3 [ 0.056017] ... event mask: 000000070000000f [ 0.057301] rcu: Hierarchical SRCU implementation. [ 0.059434] smp: Bringing up secondary CPUs ... [ 0.060579] x86: Booting SMP configuration: [ 0.061044] .... node #0, CPUs: #1 #2 #3 [ 0.066417] smp: Brought up 1 node, 4 CPUs [ 0.068019] smpboot: Max logical packages: 1 [ 0.069014] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.138567] node 0 deferred pages initialised in 66ms [ 0.141170] devtmpfs: initialized [ 0.143222] x86/mm: Memory block size: 128MB [ 0.145857] gcov: version magic: 0x41383552 [ 0.147644] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151159] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154579] pinctrl core: initialized pinctrl subsystem [ 0.157366] [ 0.157888] ************************************************************* [ 0.160021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162018] ** ** [ 0.165024] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167018] ** ** [ 0.169020] ** This means that this kernel is built to expose internal ** [ 0.171019] ** IOMMU data structures, which may compromise security on ** [ 0.174023] ** your system. ** [ 0.176020] ** ** [ 0.178095] ** If you see this message and you are not debugging the ** [ 0.181023] ** kernel, report this immediately to your vendor! ** [ 0.183022] ** ** [ 0.185023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187024] ************************************************************* [ 0.190716] NET: Registered protocol family 16 [ 0.193411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.199068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202911] cpuidle: using governor menu [ 0.205688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.209742] PCI: Using configuration type 1 for base access [ 0.211271] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.222052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224077] cryptd: max_cpu_qlen set to 1000 [ 0.226753] ACPI: Added _OSI(Module Device) [ 0.228019] ACPI: Added _OSI(Processor Device) [ 0.229015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235131] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241405] ACPI: Interpreter enabled [ 0.242064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243014] ACPI: Using IOAPIC for interrupt routing [ 0.245159] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257701] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.259057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268372] acpiphp: Slot [2] registered [ 0.269000] acpiphp: Slot [5] registered [ 0.269000] acpiphp: Slot [6] registered [ 0.269122] acpiphp: Slot [7] registered [ 0.270000] acpiphp: Slot [8] registered [ 0.271137] acpiphp: Slot [9] registered [ 0.272128] acpiphp: Slot [10] registered [ 0.273098] acpiphp: Slot [3] registered [ 0.274087] acpiphp: Slot [4] registered [ 0.275088] acpiphp: Slot [11] registered [ 0.276109] acpiphp: Slot [12] registered [ 0.278210] acpiphp: Slot [13] registered [ 0.280182] acpiphp: Slot [14] registered [ 0.282214] acpiphp: Slot [15] registered [ 0.284251] acpiphp: Slot [16] registered [ 0.286076] acpiphp: Slot [17] registered [ 0.287191] acpiphp: Slot [18] registered [ 0.289169] acpiphp: Slot [19] registered [ 0.292171] acpiphp: Slot [20] registered [ 0.293140] acpiphp: Slot [21] registered [ 0.295124] acpiphp: Slot [22] registered [ 0.296171] acpiphp: Slot [23] registered [ 0.298126] acpiphp: Slot [24] registered [ 0.299141] acpiphp: Slot [25] registered [ 0.300135] acpiphp: Slot [26] registered [ 0.302219] acpiphp: Slot [27] registered [ 0.304190] acpiphp: Slot [28] registered [ 0.305118] acpiphp: Slot [29] registered [ 0.307132] acpiphp: Slot [30] registered [ 0.308136] acpiphp: Slot [31] registered [ 0.310076] PCI host bridge to bus 0000:00 [ 0.311022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326046] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328404] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.332205] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335548] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345825] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.354100] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.357028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.359028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.364620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.367863] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.370054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.373874] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.379019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400031] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407136] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413076] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.436033] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448012] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474033] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.504000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.514018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.525020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.543017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555127] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.566020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.602518] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.640022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.652000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.661026] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704199] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.706412] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.710477] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.715246] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.720238] iommu: Default domain type: Passthrough [ 0.722518] SCSI subsystem initialized [ 0.723235] ACPI: bus type USB registered [ 0.725163] usbcore: registered new interface driver usbfs [ 0.727099] usbcore: registered new interface driver hub [ 0.730113] usbcore: registered new device driver usb [ 0.732202] pps_core: LinuxPPS API ver. 1 registered [ 0.733009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.736089] PTP clock support registered [ 0.738169] EDAC MC: Ver: 3.0.0 [ 0.741149] PCI: Using ACPI for IRQ routing [ 0.742833] NetLabel: Initializing [ 0.744013] NetLabel: domain hash size = 128 [ 0.745011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.748087] NetLabel: unlabeled traffic allowed by default [ 0.750155] vgaarb: loaded [ 0.752293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.754017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.763000] clocksource: Switched to clocksource kvm-clock [ 0.870920] VFS: Disk quotas dquot_6.6.0 [ 0.872811] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875729] *** VALIDATE ramfs *** [ 0.877549] *** VALIDATE hugetlbfs *** [ 0.879590] pnp: PnP ACPI init [ 0.882353] pnp: PnP ACPI: found 6 devices [ 0.902075] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.906030] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.908570] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.911241] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.913903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.916793] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.920255] NET: Registered protocol family 2 [ 0.922225] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.926498] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.932152] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.938739] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.942341] TCP: Hash tables configured (established 65536 bind 65536) [ 0.945375] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.948881] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951722] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.954653] NET: Registered protocol family 1 [ 0.957423] RPC: Registered named UNIX socket transport module. [ 0.959027] RPC: Registered udp transport module. [ 0.960668] RPC: Registered tcp transport module. [ 0.962478] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.965109] NET: Registered protocol family 44 [ 0.966969] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.969410] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.971673] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.974200] PCI: CLS 0 bytes, default 64 [ 0.976158] Unpacking initramfs... [ 2.423155] debug: unmapping init [mem 0xffff8b66bcc54000-0xffff8b66bffbffff] [ 2.427452] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.429269] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.431887] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.927404] Initialise system trusted keyrings [ 2.929543] Key type blacklist registered [ 2.932457] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.942138] zbud: loaded [ 2.947157] *** VALIDATE nfs *** [ 2.948657] *** VALIDATE nfs4 *** [ 2.950381] pstore: using deflate compression [ 2.954639] Platform Keyring initialized [ 3.063749] NET: Registered protocol family 38 [ 3.065636] Key type asymmetric registered [ 3.067294] Asymmetric key parser 'x509' registered [ 3.069212] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072264] io scheduler mq-deadline registered [ 3.074221] io scheduler kyber registered [ 3.075847] io scheduler bfq registered [ 3.078056] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.081367] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.084627] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.088227] ACPI: Power Button [PWRF] [ 3.094763] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.101797] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.117682] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.127315] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.142236] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.172727] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.201632] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.209546] Non-volatile memory driver v1.3 [ 3.212183] Linux agpgart interface v0.103 [ 3.251288] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.255178] vda: detected capacity change from 0 to 74862592 [ 3.272452] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.275279] vdb: detected capacity change from 0 to 1073741824 [ 3.290090] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.293063] vdc: detected capacity change from 0 to 2621440000 [ 3.307283] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.310041] vdd: detected capacity change from 0 to 2621440000 [ 3.327504] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.330387] vde: detected capacity change from 0 to 4294967296 [ 3.347099] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.351359] vdf: detected capacity change from 0 to 4294967296 [ 3.363355] libphy: Fixed MDIO Bus: probed [ 3.369378] usbcore: registered new interface driver usbserial_generic [ 3.372532] usbserial: USB Serial support registered for generic [ 3.375339] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.380192] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.382484] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.386573] mousedev: PS/2 mouse device common for all mice [ 3.391417] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.391791] rtc_cmos 00:05: RTC can wake from S4 [ 3.399537] rtc_cmos 00:05: registered as rtc0 [ 3.401740] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.401765] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.403753] intel_pstate: CPU model not supported [ 3.411185] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415728] hid: raw HID events driver (C) Jiri Kosina [ 3.418595] usbcore: registered new interface driver usbhid [ 3.420518] usbhid: USB HID core driver [ 3.421684] drop_monitor: Initializing network drop monitor service [ 3.423776] Initializing XFRM netlink socket [ 3.426676] NET: Registered protocol family 10 [ 3.430444] Segment Routing with IPv6 [ 3.432226] NET: Registered protocol family 17 [ 3.435182] mpls_gso: MPLS GSO support [ 3.441910] RAS: Correctable Errors collector initialized. [ 3.444225] AVX version of gcm_enc/dec engaged. [ 3.446116] AES CTR mode by8 optimization enabled [ 3.538556] sched_clock: Marking stable (3538499811, 0)->(4527506734, -989006923) [ 3.542437] registered taskstats version 1 [ 3.544470] Loading compiled-in X.509 certificates [ 3.546626] zswap: loaded using pool lzo/zbud [ 3.572305] Key type big_key registered [ 3.587191] Key type encrypted registered [ 3.589296] ima: No TPM chip found, activating TPM-bypass! [ 3.592695] ima: Allocated hash algorithm: sha1 [ 3.594353] ima: No architecture policies found [ 3.596092] evm: Initialising EVM extended attributes: [ 3.598205] evm: security.selinux [ 3.599284] evm: security.ima [ 3.600375] evm: security.capability [ 3.601748] evm: HMAC attrs: 0x1 [ 3.604110] rtc_cmos 00:05: setting system clock to 2026-07-28 06:30:05 UTC (1785220205) [ 3.610896] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 3.614167] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 3.628190] Write protecting the kernel read-only data: 28672k [ 3.632788] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 3.636534] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 3.675612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.684780] systemd[1]: Detected virtualization kvm. [ 3.686796] systemd[1]: Detected architecture x86-64. [ 3.688909] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.714934] systemd[1]: No hostname configured. [ 3.716830] systemd[1]: Set hostname to . [ 3.718992] random: systemd: uninitialized urandom read (16 bytes read) [ 3.721379] systemd[1]: Initializing machine ID from random generator. [ 3.799850] random: ln: uninitialized urandom read (6 bytes read) [ 3.888536] random: systemd: uninitialized urandom read (16 bytes read) [ 3.891259] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.895367] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.921072] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.603275] device-mapper: uevent: version 1.0.3 [ 4.605868] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.443408] virtio_net virtio0 ens2: renamed from eth0 [ 5.445898] random: fast init done [ 5.506712] scsi host0: ata_piix [ 5.532938] scsi host1: ata_piix [ 5.534536] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.536570] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.457591] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.112328] random: crng init done [ 10.113956] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.664378] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.103386] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.149830] SELinux: Disabled at runtime. [ 13.294431] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.317463] systemd[1]: Detected virtualization kvm. [ 13.323465] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.034756] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.041223] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.060667] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.070204] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.079314] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.098668] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.113708] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ 15.218318] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.451055] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.386740] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.605096] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.034789] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.154175] EDAC sbridge: Ver: 1.1.2 [ 21.917581] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 22.798682] NFS: Registering the id_resolver key type [ 22.801057] Key type id_resolver registered [ 22.803628] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg407-server login: [ 57.038014] hrtimer: interrupt took 7551805 ns [ 86.979628] libcfs: loading out-of-tree module taints kernel. [ 87.053458] Key type ._llcrypt registered [ 87.058196] Key type .llcrypt registered [ 87.170730] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 109.376543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 111.307558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.326145] alg: No test for adler32 (adler32-zlib) [ 112.953922] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 114.287233] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 116.079176] Key type lgssc registered [ 118.429127] Lustre: Echo OBD driver; http://www.lustre.org/ [ 137.316034] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 186.224933] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 200.970852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 200.997554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.232219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 202.286683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 202.464855] Lustre: lustre-MDT0000: new disk, initializing [ 202.580102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.594900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.852146] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.558411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.704599] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.731895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 222.742224] Lustre: Skipped 1 previous similar message [ 222.894183] Lustre: lustre-MDT0001: new disk, initializing [ 222.981302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 223.005034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 223.015439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 227.318515] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.297220] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 242.507581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 242.832342] Lustre: lustre-OST0000: new disk, initializing [ 242.835895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 242.848984] Lustre: 8460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.957188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 243.816996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.832752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 243.947407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 249.497863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.088708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.249105] Lustre: lustre-OST0001: new disk, initializing [ 265.263489] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.272464] Lustre: 9532:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.413468] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 272.288778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 275.021043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.033619] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.120882] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.868312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 290.480279] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.499850] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.426576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing yml_node [ 306.932324] Lustre: DEBUG MARKER: Client: 2.17.54.204 [ 309.868830] Lustre: DEBUG MARKER: MDS: 2.17.54.204 [ 312.133576] Lustre: DEBUG MARKER: OSS: 2.17.54.204 [ 313.702402] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jul 28 02:35:13 EDT 2026 [ 332.457532] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 334.387994] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 351.203573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 351.211779] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 351.234161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.224794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.226384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.248865] Lustre: Skipped 2 previous similar messages [ 352.264811] Lustre: Skipped 2 previous similar messages [ 356.494359] Lustre: server umount lustre-MDT0000 complete [ 360.259460] LustreError: 9543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785220562 with bad export cookie 14262968941888727066 [ 360.278463] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 360.282791] LustreError: 9543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 362.470088] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.471012] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.476345] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.476354] Lustre: Skipped 1 previous similar message [ 362.487668] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 362.523715] Lustre: Skipped 1 previous similar message [ 365.471677] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.481180] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.505536] Lustre: Skipped 2 previous similar messages [ 366.706964] Lustre: server umount lustre-MDT0001 complete [ 370.208664] Lustre: server umount lustre-OST0000 complete [ 373.605361] Lustre: server umount lustre-OST0001 complete [ 380.137129] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 390.149305] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 426.963513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.208322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 427.236426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 427.343460] Lustre: lustre-MDT0000: new disk, initializing [ 427.412467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.424888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 431.944968] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 442.475899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.575903] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.655304] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.664621] Lustre: Skipped 1 previous similar message [ 442.733157] Lustre: lustre-MDT0001: new disk, initializing [ 442.841063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.895270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.904423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.024582] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 457.333737] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 463.425724] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 469.715373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 470.001433] Lustre: lustre-OST0000: new disk, initializing [ 470.006260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 470.012225] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 470.095450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.047761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 472.057664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 472.142090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.738280] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 485.132065] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 489.205933] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 489.408168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 493.294906] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.500742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 497.642217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 497.643346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.651590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 497.680451] Lustre: Skipped 1 previous similar message [ 501.509245] Lustre: server umount lustre-OST0000 complete [ 507.881367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.899876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 507.912171] Lustre: Skipped 2 previous similar messages [ 511.447610] Lustre: server umount lustre-MDT0000 complete [ 512.993509] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 513.008427] LustreError: 15091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 515.394068] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785220717 with bad export cookie 14262968941888728592 [ 515.412203] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.415927] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 515.811968] Lustre: server umount lustre-MDT0001 complete [ 520.360830] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 02:38:40 (1785220720) [ 522.838927] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 524.888420] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 02:38:44 (1785220724) [ 531.851277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.258762] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.363309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.865346] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 537.572238] LustreError: 19464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.809898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.463140] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 557.418575] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.262845] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 569.682979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.831501] LustreError: 21282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.855420] LustreError: 21282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 576.084984] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 583.336165] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 590.830016] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 590.844222] LustreError: Skipped 1 previous similar message [ 590.847236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.859354] Lustre: Skipped 1 previous similar message [ 590.863684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.873092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 596.710966] Lustre: server umount lustre-OST0000 complete [ 601.568484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 614.367159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 614.571705] Lustre: server umount lustre-MDT0000 complete [ 617.442185] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.471294] LustreError: 19460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 618.309244] LustreError: 20198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785220820 with bad export cookie 14262968941888729474 [ 618.313763] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.334288] LustreError: 20198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 618.688284] Lustre: server umount lustre-MDT0001 complete [ 629.123968] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 631.693266] Key type lgssc unregistered [ 632.004310] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.019106] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.041713] LNet: Removed LNI 192.168.204.107@tcp [ 632.911333] Key type .llcrypt unregistered [ 632.916575] Key type ._llcrypt unregistered [ 653.211797] Key type ._llcrypt registered [ 653.213991] Key type .llcrypt registered [ 653.287418] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 654.142577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.189262] alg: No test for adler32 (adler32-zlib) [ 655.254878] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 655.506601] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 657.152593] Key type lgssc registered [ 658.094937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 664.818231] loop: module loaded [ 664.891653] loop0: detected capacity change from 0 to 204800000 [ 665.132940] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.151386] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 665.546842] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.197604] loop0: detected capacity change from 0 to 204800000 [ 670.350229] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.357436] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 670.365521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 674.951968] loop0: detected capacity change from 0 to 204800000 [ 679.480990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 679.500503] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.758884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 680.805121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 680.870206] Lustre: lustre-MDT0000: new disk, initializing [ 680.945043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.959184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 685.287739] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 694.082748] loop1: detected capacity change from 0 to 204800000 [ 700.096574] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 700.271544] Lustre: lustre-OST07c6: new disk, initializing [ 700.273710] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 700.277239] Lustre: Skipped 1 previous similar message [ 700.286881] Lustre: 26183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 700.384583] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 710.067734] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 710.302216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 710.309452] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 710.355506] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 716.125704] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 716.156562] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 716.177345] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 716.231719] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 716.252839] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 716.274691] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 727.882384] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 727.894223] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 727.905186] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 727.916546] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 727.947835] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 727.956090] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.922900] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 732.928750] Lustre: 25172:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 732.934314] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 732.939052] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.943407] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 732.949901] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.954427] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 732.959228] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.963713] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 732.968126] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 732.972257] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 732.976042] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.168965] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 736.176966] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 736.181801] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 736.186286] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.189152] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 736.202027] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.206173] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 736.213133] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.218726] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 736.224168] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 736.231334] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 736.240980] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.240527] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 741.252417] Lustre: 25170:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 741.260922] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 741.270990] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.283859] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 741.289159] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.294566] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 741.302691] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.309287] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 741.319844] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 741.330805] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 741.343068] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.230541] Lustre: 26875:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 752.251384] Lustre: 26875:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 752.255474] Lustre: 26875:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 752.265039] Lustre: 26875:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.276386] Lustre: 26875:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.284504] Lustre: 26875:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.293438] Lustre: 26875:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 752.303707] Lustre: 26875:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.317458] Lustre: 26875:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.334895] Lustre: 26875:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 752.349600] Lustre: 26875:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.366964] Lustre: 26875:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 802.276310] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.288977] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 805.746254] Lustre: server umount lustre-OST07c6 complete [ 813.549852] Lustre: server umount lustre-MDT0000 complete [ 826.918765] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 828.970245] Key type lgssc unregistered [ 829.181342] LNet: 28737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.186918] LNetError: 28737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.204518] LNet: Removed LNI 192.168.204.107@tcp [ 829.806937] Key type .llcrypt unregistered [ 829.811473] Key type ._llcrypt unregistered [ 845.217840] Key type ._llcrypt registered [ 845.221577] Key type .llcrypt registered [ 845.305297] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 846.266598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.288499] alg: No test for adler32 (adler32-zlib) [ 847.293580] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 847.521517] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 849.192109] Key type lgssc registered [ 850.200867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 856.838292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 856.851794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.303715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.205440] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 868.835047] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.859349] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 869.573611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.909294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.011694] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 880.531740] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 885.506658] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 891.305746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 891.555354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.929551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 903.853897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 904.150687] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 919.522732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 919.527384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 919.540689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 919.550873] Lustre: Skipped 1 previous similar message [ 924.571577] Lustre: server umount lustre-OST0000 complete [ 929.762679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 929.766623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 929.772215] Lustre: Skipped 1 previous similar message [ 929.775860] Lustre: Skipped 1 previous similar message [ 934.260886] Lustre: server umount lustre-MDT0000 complete [ 934.881469] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 934.903121] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 936.948535] LustreError: 29968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221138 with bad export cookie 116028279203516292 [ 936.955979] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 936.958803] LustreError: 29968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 937.172399] Lustre: server umount lustre-MDT0001 complete [ 946.444687] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 949.057690] Key type lgssc unregistered [ 949.424691] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.443258] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.466059] LNet: Removed LNI 192.168.204.107@tcp [ 950.280345] Key type .llcrypt unregistered [ 950.282747] Key type ._llcrypt unregistered [ 967.971382] Key type ._llcrypt registered [ 967.973654] Key type .llcrypt registered [ 968.058304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 968.685197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 968.746632] alg: No test for adler32 (adler32-zlib) [ 969.799614] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 970.046040] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 971.720051] Key type lgssc registered [ 972.634853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.855603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.879756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.280236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 983.861480] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 990.689066] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.701614] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 991.029734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.370528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.089962] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1001.751375] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1006.192485] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1011.937134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.229136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.644763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1022.175298] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1025.596071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1035.760227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1035.761549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.782177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1035.786387] Lustre: Skipped 1 previous similar message [ 1040.877066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.885978] Lustre: Skipped 1 previous similar message [ 1040.993352] Lustre: server umount lustre-OST0000 complete [ 1045.986739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1046.000477] Lustre: Skipped 1 previous similar message [ 1046.005367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1046.016025] Lustre: Skipped 1 previous similar message [ 1051.106632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.113546] Lustre: Skipped 1 previous similar message [ 1056.227065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1056.231744] Lustre: Skipped 1 previous similar message [ 1058.271126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.442865] Lustre: server umount lustre-MDT0000 complete [ 1061.351728] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.372027] LustreError: 34997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1061.462029] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221263 with bad export cookie 12352623864338852959 [ 1061.472518] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1061.473437] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1061.832145] Lustre: server umount lustre-MDT0001 complete [ 1070.949326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1073.143493] Key type lgssc unregistered [ 1073.360102] LNet: 38555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.370189] LNetError: 38555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.387742] LNet: Removed LNI 192.168.204.107@tcp [ 1074.032446] Key type .llcrypt unregistered [ 1074.034145] Key type ._llcrypt unregistered [ 1094.596182] Key type ._llcrypt registered [ 1094.599479] Key type .llcrypt registered [ 1094.690479] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1095.297480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.372297] alg: No test for adler32 (adler32-zlib) [ 1096.418539] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1096.616855] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1098.256739] Key type lgssc registered [ 1099.172818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1105.533111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1105.557903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.003355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.366699] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1116.134800] LustreError: 39975:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.155287] LustreError: 39975:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1117.122442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.398582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.798852] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1126.359892] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1130.140279] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1135.231524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1135.457476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.189264] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1146.144879] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1151.862687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1157.092729] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.093866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1157.099823] Lustre: Skipped 2 previous similar messages [ 1157.104217] Lustre: Skipped 1 previous similar message [ 1158.299536] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1158.538501] Lustre: server umount lustre-MDT0000 complete [ 1160.923873] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221362 with bad export cookie 342708438426509573 [ 1160.926817] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1160.946453] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.208062] LustreError: 39970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.213422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1162.226501] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1163.201427] Lustre: server umount lustre-MDT0001 complete [ 1174.805604] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1785221376 with bad export cookie 342708438426510238 [ 1174.816668] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1182.855408] Lustre: server umount lustre-OST0000 complete [ 1194.042650] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1195.931957] Key type lgssc unregistered [ 1196.146892] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.152840] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.166306] LNet: Removed LNI 192.168.204.107@tcp [ 1196.780741] Key type .llcrypt unregistered [ 1196.785371] Key type ._llcrypt unregistered [ 1213.227411] Key type ._llcrypt registered [ 1213.229654] Key type .llcrypt registered [ 1213.297463] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1214.028561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.089529] alg: No test for adler32 (adler32-zlib) [ 1215.053525] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1215.210379] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1216.831258] Key type lgssc registered [ 1217.710255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.443254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.456969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.766276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.077425] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1234.611298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.870952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.755939] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1242.746479] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.704644] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1251.368382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1251.628798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.674550] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1261.338662] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1271.092827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1276.389844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1276.390189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1276.410343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1276.421786] Lustre: Skipped 1 previous similar message [ 1276.433027] Lustre: Skipped 1 previous similar message [ 1277.095370] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1281.505849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1281.510601] Lustre: Skipped 1 previous similar message [ 1282.407280] Lustre: server umount lustre-OST0000 complete [ 1301.474549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.479733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.486587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.489093] Lustre: Skipped 1 previous similar message [ 1307.106540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.121025] Lustre: Skipped 2 previous similar messages [ 1312.234121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1312.324899] Lustre: server umount lustre-MDT0000 complete [ 1314.713840] LustreError: 45209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221516 with bad export cookie 16325050164714275816 [ 1314.719138] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.725897] LustreError: 45209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1329.119085] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1329.275273] Lustre: server umount lustre-MDT0001 complete [ 1336.765811] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1338.660947] Key type lgssc unregistered [ 1338.932189] LNet: 48958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.940801] LNetError: 48958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.961128] LNet: Removed LNI 192.168.204.107@tcp [ 1339.694290] Key type .llcrypt unregistered [ 1339.697206] Key type ._llcrypt unregistered [ 1356.172248] Key type ._llcrypt registered [ 1356.175345] Key type .llcrypt registered [ 1356.256488] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1356.806405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1356.819322] alg: No test for adler32 (adler32-zlib) [ 1357.796365] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1357.964779] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1359.583177] Key type lgssc registered [ 1360.390883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1365.261920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1365.278319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.525370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.103943] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1374.491675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.720426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1377.611709] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1382.244365] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.106381] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1390.602902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1390.832973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.849947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1394.867445] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1400.268209] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1404.591811] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1407.265336] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1418.830775] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-28 2h53m38s [ 1419.868176] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1422.489817] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1423.559031] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 1425.890121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1425.890786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.895880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1425.903340] Lustre: Skipped 1 previous similar message [ 1427.543611] Lustre: server umount lustre-MDT0000 complete [ 1428.618415] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.629800] LustreError: 50351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1429.930994] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1433.072612] LustreError: 50356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1433.088345] LustreError: 50356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1434.591827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.691866] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1434.761106] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (not set up) [ 1434.765199] Lustre: Skipped 5 previous similar messages [ 1434.865920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.910176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1435.882546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1437.813810] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1440.350889] Lustre: DEBUG MARKER: Wait for df (36639) ... [ 1441.463847] Lustre: DEBUG MARKER: done [ 1445.345294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.355756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.365092] Lustre: Skipped 1 previous similar message [ 1445.369715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.374579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1446.374642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.302322] Lustre: server umount lustre-OST0000 complete [ 1456.609719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.612629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1456.616515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.616524] Lustre: Skipped 2 previous similar messages [ 1456.625841] Lustre: Skipped 1 previous similar message [ 1458.712930] Lustre: server umount lustre-MDT0000 complete [ 1461.203858] LustreError: 50336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221663 with bad export cookie 12784513949267558283 [ 1461.208972] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.212084] LustreError: 50336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1461.428044] Lustre: server umount lustre-MDT0001 complete [ 1467.575337] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1469.117220] Key type lgssc unregistered [ 1469.276979] LNet: 55267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.281682] LNetError: 55267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.296811] LNet: Removed LNI 192.168.204.107@tcp [ 1469.790822] Key type .llcrypt unregistered [ 1469.792333] Key type ._llcrypt unregistered [ 1494.640754] Key type ._llcrypt registered [ 1494.642531] Key type .llcrypt registered [ 1494.708338] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1495.296533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.319758] alg: No test for adler32 (adler32-zlib) [ 1496.240761] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1496.388569] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1497.999301] Key type lgssc registered [ 1498.648184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1503.289395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1503.300773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.481769] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1504.498175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1504.585856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1506.919901] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1511.786668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.854336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1511.881933] Lustre: 57453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1512.035462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1514.397969] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1518.691837] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1522.124506] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1525.877124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.027882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1526.146100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1529.658936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1533.492654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1533.951164] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1538.394786] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-28 2h55m38s [ 1539.390878] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1543.460758] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1544.853607] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1544.882798] LustreError: 58204:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1565.357178] Lustre: lustre-MDT0000: Client 6793fb87-e79b-45a0-800b-ce580ddf1f12 (at 192.168.204.7@tcp) reconnecting [ 1584.927306] LustreError: 58204:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1586.302523] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1587.768964] Lustre: DEBUG MARKER: done [ 1594.850332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.852641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.853996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.856818] LustreError: Skipped 1 previous similar message [ 1594.878766] Lustre: Skipped 1 previous similar message [ 1597.373747] Lustre: server umount lustre-OST0000 complete [ 1599.969167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1599.970717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1599.983393] Lustre: Skipped 1 previous similar message [ 1599.988793] Lustre: Skipped 1 previous similar message [ 1601.057139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.064263] Lustre: Skipped 3 previous similar messages [ 1605.091220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1605.100663] Lustre: Skipped 1 previous similar message [ 1605.804364] Lustre: server umount lustre-MDT0000 complete [ 1608.201439] LustreError: 58193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221810 with bad export cookie 5692165641994270049 [ 1608.203870] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1608.207138] LustreError: 58193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1608.373901] Lustre: server umount lustre-MDT0001 complete [ 1614.615383] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1616.219316] Key type lgssc unregistered [ 1616.394102] LNet: 61777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.401349] LNetError: 61777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.417234] LNet: Removed LNI 192.168.204.107@tcp [ 1616.921948] Key type .llcrypt unregistered [ 1616.923966] Key type ._llcrypt unregistered [ 1643.412918] Key type ._llcrypt registered [ 1643.414592] Key type .llcrypt registered [ 1643.478246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1643.914562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1643.922889] alg: No test for adler32 (adler32-zlib) [ 1644.838439] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1644.992359] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1646.607449] Key type lgssc registered [ 1647.375859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1649.500487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1649.515753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.706786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1650.722555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1650.845376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1651.452451] Lustre: Failing over lustre-MDT0000 [ 1651.661781] Lustre: server umount lustre-MDT0000 complete [ 1656.046500] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:57:36 (1785221856) [ 1660.585623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.909182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.462438] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1668.815760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1668.884443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1668.903343] Lustre: 64551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1669.057678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1671.919730] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1676.618902] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.954714] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1684.406277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1684.537144] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1684.732110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.321163] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1690.102810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1693.271867] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1698.324095] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1704.622373] Lustre: server umount lustre-MDT0000 complete [ 1704.929968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1704.939168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.953380] LustreError: 64562:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.967107] LustreError: 64562:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1705.441602] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1706.841707] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221908 with bad export cookie 93897812672814966 [ 1706.844423] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1706.848318] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1707.026220] Lustre: server umount lustre-MDT0001 complete [ 1709.022960] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1711.785424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1718.024401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.943561] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221912/real 1785221912] req@0000000070d4ea7c x1871940782425344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785221928 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1726.976226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.997228] Lustre: Skipped 1 previous similar message [ 1727.967919] Lustre: 63477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221913/real 1785221913] req@00000000014a20e7 x1871940782425472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785221929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1731.123389] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x14d9790298ca859 [ 1731.174529] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 1731.735459] LustreError: 66398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.789043] LustreError: 66398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1731.885532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.133268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1732.898572] LustreError: 66399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.900282] Lustre: 63476:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221918/real 1785221918] req@00000000b8151218 x1871940782425856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785221934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1736.995127] LustreError: 68318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1737.040246] LustreError: 68318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1738.016984] Lustre: 63477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221923/real 1785221923] req@00000000bc89c3a6 x1871940782425984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785221939 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1738.061381] Lustre: 63477:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1739.408037] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1742.119935] Lustre: 63478:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785221927/real 1785221927] req@00000000a7b4700b x1871940782426240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785221943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1742.314568] LustreError: 66398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.338539] LustreError: 66398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1745.387428] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1750.497037] LustreError: 68318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1750.535434] LustreError: 68318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1753.949050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.578054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1759.726449] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1761.709613] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1770.904526] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1778.617587] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1790.436329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.443898] Lustre: Skipped 1 previous similar message [ 1790.444173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.458681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1790.469850] Lustre: Skipped 2 previous similar messages [ 1792.213065] Lustre: server umount lustre-MDT0000 complete [ 1795.559735] LustreError: 68320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1795.602357] LustreError: 68320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1797.929473] LustreError: 64543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785221999 with bad export cookie 93897812672817241 [ 1797.934442] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.956545] LustreError: 64543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1798.562696] Lustre: server umount lustre-MDT0001 complete [ 1810.475894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1817.076544] Lustre: 63477:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785222002/real 1785222002] req@00000000cc15c3bf x1871940782455808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785222018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1817.137154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1817.149730] Lustre: Skipped 1 previous similar message [ 1822.440409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.783981] LustreError: 63474:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002b27754e x1871940782457088/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1843.093448] LustreError: 66398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1843.220421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.268878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1845.288402] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1847.080322] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1855.352410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.258552] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 1862.127977] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1867.656659] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1873.186206] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1879.514498] Lustre: server umount lustre-MDT0000 complete [ 1881.570052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1881.576863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.981859] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222084 with bad export cookie 93897812672818620 [ 1882.985192] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.995611] LustreError: 64542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1885.392285] Lustre: server umount lustre-MDT0001 complete [ 1890.468713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1893.462754] Lustre: DEBUG MARKER: files compared the same [ 1905.680789] Lustre: server umount lustre-OST0000 complete [ 1920.102500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 1922.508791] Key type lgssc unregistered [ 1922.777441] LNet: 74295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.787638] LNetError: 74295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.809357] LNet: Removed LNI 192.168.204.107@tcp [ 1923.541221] Key type .llcrypt unregistered [ 1923.547786] Key type ._llcrypt unregistered [ 1942.843584] Key type ._llcrypt registered [ 1942.847890] Key type .llcrypt registered [ 1942.935748] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 1943.641473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.746424] alg: No test for adler32 (adler32-zlib) [ 1944.820747] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 1945.052419] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 1946.743209] Key type lgssc registered [ 1947.594820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.360256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1954.374259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.746998] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.776428] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 1966.059920] LustreError: 75689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.074078] LustreError: 75689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1967.196539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.462696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1971.715570] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 1978.163601] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1983.394722] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1989.228379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.449048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.235859] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug +malloc all [ 2000.751574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2002.282169] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2011.103990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2011.116189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.117030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2011.130664] Lustre: Skipped 1 previous similar message [ 2015.241891] Lustre: server umount lustre-OST0000 complete [ 2018.784753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2018.799390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2018.816722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2018.829379] Lustre: Skipped 1 previous similar message [ 2021.344215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2021.345287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2024.887274] Lustre: server umount lustre-MDT0000 complete [ 2026.465235] LustreError: 75689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.501489] LustreError: 75689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2028.184890] LustreError: 75668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222230 with bad export cookie 5875263858614591434 [ 2028.194420] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2028.481290] Lustre: server umount lustre-MDT0001 complete [ 2038.461427] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2040.798822] Key type lgssc unregistered [ 2041.067544] LNet: 79217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.073197] LNetError: 79217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.088619] LNet: Removed LNI 192.168.204.107@tcp [ 2041.732161] Key type .llcrypt unregistered [ 2041.734527] Key type ._llcrypt unregistered [ 2061.094946] Key type ._llcrypt registered [ 2061.098744] Key type .llcrypt registered [ 2061.316612] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2062.422787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.465765] alg: No test for adler32 (adler32-zlib) [ 2063.543456] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2063.781465] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2065.471392] Key type lgssc registered [ 2066.386893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2074.810892] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2074.843786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.463606] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2095.583558] LustreError: 80587:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [ 2122.271213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2128.474583] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2136.454789] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2144.067278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.142185] Lustre: *** cfs_fail_loc=706, val=0*** [ 2144.156677] LustreError: 79924:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000baaaddca x1871941221746944/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2144.485744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.598017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2149.218312] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2149.856642] LustreError: 80612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.894813] LustreError: 80612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2154.984411] LustreError: 80613:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.030379] LustreError: 81674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.041268] LustreError: 81674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2158.636974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.968254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.384653] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2170.264889] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.249514] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.557846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.571283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.676321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.682374] Lustre: Skipped 2 previous similar messages [ 2189.795498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.809912] Lustre: Skipped 1 previous similar message [ 2191.327135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2191.421183] Lustre: server umount lustre-OST0000 complete [ 2200.036393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.048775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2200.048955] Lustre: Skipped 1 previous similar message [ 2200.051045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.705553] Lustre: server umount lustre-MDT0000 complete [ 2205.152235] LustreError: 81678:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.166976] LustreError: 81678:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2205.773744] LustreError: 80594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222407 with bad export cookie 12481352464855260866 [ 2205.783539] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.792044] LustreError: 80594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2206.270057] Lustre: server umount lustre-MDT0001 complete [ 2218.174446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2221.131566] Key type lgssc unregistered [ 2221.427749] LNet: 84204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.440578] LNetError: 84204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.458537] LNet: Removed LNI 192.168.204.107@tcp [ 2222.411787] Key type .llcrypt unregistered [ 2222.417241] Key type ._llcrypt unregistered [ 2247.946288] Key type ._llcrypt registered [ 2247.948706] Key type .llcrypt registered [ 2248.052386] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2249.014271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.066070] alg: No test for adler32 (adler32-zlib) [ 2250.109684] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2250.402084] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2252.063141] Key type lgssc registered [ 2253.117939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2260.558588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2260.570666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2265.664150] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2272.902551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.458979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.090747] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2278.882443] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2278.917527] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2284.001735] LustreError: 86312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.014893] LustreError: 86312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2286.229738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.549778] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.565429] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2286.565521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.690481] LustreError: 87056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.719777] LustreError: 87056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2292.825424] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2296.824884] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.855687] LustreError: 86309:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2301.905380] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.917716] Lustre: MGS: Client 72dd0840-072f-4fca-b155-70285e1461fd (at 0@lo) reconnecting [ 2301.935458] LustreError: 86323:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2302.255956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.343799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2307.899410] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2322.920869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.925159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2322.943586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2322.954132] Lustre: Skipped 1 previous similar message [ 2328.037972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2328.049139] Lustre: Skipped 1 previous similar message [ 2328.818497] Lustre: server umount lustre-OST0000 complete [ 2333.165475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.168972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2333.169651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.181387] Lustre: Skipped 2 previous similar messages [ 2333.201885] Lustre: Skipped 1 previous similar message [ 2338.273177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.284492] Lustre: Skipped 1 previous similar message [ 2338.429661] Lustre: server umount lustre-MDT0000 complete [ 2342.613269] Lustre: server umount lustre-MDT0001 complete [ 2346.187780] Lustre: server umount MGS complete [ 2359.020865] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:09:18 (1785222558) [ 2385.276100] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 2394.381278] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2425.198320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.678597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2436.796097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.977358] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2437.011276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2437.072204] Lustre: lustre-MDT0001: new disk, initializing [ 2437.133382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2437.176312] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2437.186312] Lustre: 93498:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2437.195689] LustreError: 93498:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2441.420844] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2451.133291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2451.301850] Lustre: lustre-OST0000: new disk, initializing [ 2451.305148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2451.312531] Lustre: 94265:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2451.380207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.706603] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2460.707697] Lustre: 94740:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000dd51a80 x1871941417869568/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785222678 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2465.457769] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2470.808943] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.821857] Lustre: MGS: Client f77e2563-8463-4ad2-a8f8-e8579e71437c (at 0@lo) reconnecting [ 2470.866150] Lustre: 92765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2470.881864] LustreError: 92765:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2470.889125] LustreError: 92765:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2470.910111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2470.968911] Lustre: lustre-MDT0000: new disk, initializing [ 2471.069639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.097310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2471.183349] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2472.161243] Lustre: 95333:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a7c1a654 x1871941417876096/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785222690 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2474.987308] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2481.127114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2481.131646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2481.343597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2505.201548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2505.210209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.401801] Lustre: server umount lustre-OST0000 complete [ 2515.427921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2515.428520] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.440135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2515.445813] Lustre: Skipped 2 previous similar messages [ 2515.455225] Lustre: Skipped 2 previous similar messages [ 2518.738543] Lustre: server umount lustre-MDT0000 complete [ 2520.545540] LustreError: 93469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2520.555166] LustreError: 93469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2527.909793] Lustre: server umount lustre-MDT0001 complete [ 2530.726646] Lustre: server umount MGS complete [ 2539.577302] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:12:19 (1785222739) [ 2554.063116] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2556.419969] Key type lgssc unregistered [ 2556.682086] LNet: 97829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.686199] LNetError: 97829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.698723] LNet: Removed LNI 192.168.204.107@tcp [ 2557.304808] Key type .llcrypt unregistered [ 2557.308780] Key type ._llcrypt unregistered [ 2570.919146] Key type ._llcrypt registered [ 2570.920825] Key type .llcrypt registered [ 2571.000446] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2571.894256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2571.931048] alg: No test for adler32 (adler32-zlib) [ 2572.898621] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2573.094332] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2574.735268] Key type lgssc registered [ 2575.719709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.940168] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.959583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.254330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.981646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2593.771198] LustreError: 99031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.793951] LustreError: 99031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2594.201403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2594.542277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.708804] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2603.046533] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.471381] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2612.649063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.903032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2617.523097] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2623.616527] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2629.293170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2639.843336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.846554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.861357] Lustre: Skipped 1 previous similar message [ 2639.875735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.969641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2644.974774] Lustre: Skipped 1 previous similar message [ 2645.915712] Lustre: server umount lustre-OST0000 complete [ 2650.087064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.088696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.100238] Lustre: Skipped 1 previous similar message [ 2650.120585] Lustre: Skipped 2 previous similar messages [ 2655.201520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.206726] Lustre: Skipped 1 previous similar message [ 2660.320811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2660.325712] Lustre: Skipped 1 previous similar message [ 2663.391134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2663.511244] Lustre: server umount lustre-MDT0000 complete [ 2665.440404] LustreError: 99026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.456968] LustreError: 99026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2666.180672] LustreError: 99011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222868 with bad export cookie 7359113145723721027 [ 2666.182654] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.189932] LustreError: 99011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2666.408426] Lustre: server umount lustre-MDT0001 complete [ 2673.555908] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2675.510570] Key type lgssc unregistered [ 2675.758981] LNet: 102584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.764113] LNetError: 102584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.779510] LNet: Removed LNI 192.168.204.107@tcp [ 2676.352469] Key type .llcrypt unregistered [ 2676.354963] Key type ._llcrypt unregistered [ 2687.901772] Key type ._llcrypt registered [ 2687.905153] Key type .llcrypt registered [ 2687.980404] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2688.588885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.637378] alg: No test for adler32 (adler32-zlib) [ 2689.622965] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2689.798692] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2691.423215] Key type lgssc registered [ 2692.287925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2696.667606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2696.671584] LustreError: 103643:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2696.691871] LustreError: 103647:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2696.697125] LustreError: 103643:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 2696.702153] LustreError: 103643:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2696.710720] LustreError: 103643:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2696.718549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.117659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.252940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.473138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2706.299176] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2709.947142] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lsmod [ 2711.728731] LustreError: 104850:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2711.733131] LustreError: 104851:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2711.739666] LustreError: 104850:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 2711.744129] LustreError: 104850:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2711.750569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2711.752996] LustreError: 104850:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2712.032531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.041849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.051685] Lustre: Skipped 1 previous similar message [ 2715.056367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.070664] Lustre: Skipped 2 previous similar messages [ 2720.220049] Lustre: server umount lustre-MDT0000 complete [ 2720.226603] LustreError: 103679:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2720.247019] LustreError: 103679:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2723.031793] LustreError: 103663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222924 with bad export cookie 12208945012686860798 [ 2723.037601] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.044493] LustreError: 103663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2723.255294] LustreError: 104115:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e82b073f x1871941878171392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2723.360234] Lustre: server umount lustre-MDT0001 complete [ 2735.491143] Lustre: server umount lustre-OST0000 complete [ 2741.793435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2742.102683] LustreError: 105917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.153625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.006539] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2747.364226] LustreError: 105918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.882168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.154397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2753.606618] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2756.982304] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2760.134898] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2764.270488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2764.488532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2767.322480] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2767.599896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2771.069955] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2781.153237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2781.157341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2781.163132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2781.173418] Lustre: Skipped 2 previous similar messages [ 2782.662805] Lustre: server umount lustre-OST0000 complete [ 2786.276535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.282648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2786.288824] Lustre: Skipped 1 previous similar message [ 2786.290755] Lustre: Skipped 1 previous similar message [ 2791.397110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.400975] Lustre: Skipped 2 previous similar messages [ 2793.804437] Lustre: server umount lustre-MDT0000 complete [ 2796.394231] LustreError: 105898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785222998 with bad export cookie 12208945012686861498 [ 2796.398119] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2796.402498] LustreError: 105898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2796.664383] Lustre: server umount lustre-MDT0001 complete [ 2805.373551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2807.602698] Key type lgssc unregistered [ 2807.847419] LNet: 109444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2807.859612] LNetError: 109444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2807.884535] LNet: Removed LNI 192.168.204.107@tcp [ 2808.486862] Key type .llcrypt unregistered [ 2808.488153] Key type ._llcrypt unregistered [ 2850.238787] Key type ._llcrypt registered [ 2850.242467] Key type .llcrypt registered [ 2850.441596] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 2852.413745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2852.494706] alg: No test for adler32 (adler32-zlib) [ 2854.646325] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 2855.081825] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 2856.881186] Key type lgssc registered [ 2858.604768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2869.407901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2869.428828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2871.060108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.451237] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2880.487871] LustreError: 110889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2880.498864] LustreError: 110889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2885.607925] LustreError: 110890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2887.445534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.922664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2894.310624] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2903.564169] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2909.989745] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2917.303536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.671899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.731555] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 2931.618113] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2940.220909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2947.516623] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2951.685408] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2955.753911] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2955.755261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2955.757878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2955.796973] Lustre: Skipped 1 previous similar message [ 2957.743242] Lustre: server umount lustre-OST0000 complete [ 2964.964258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.976285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.985878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.991420] Lustre: Skipped 1 previous similar message [ 2965.993373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2965.994041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.027082] Lustre: Skipped 1 previous similar message [ 2967.917776] Lustre: server umount lustre-MDT0000 complete [ 2971.104421] LustreError: 111819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.126155] LustreError: 111819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2972.160993] LustreError: 110872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223174 with bad export cookie 11248363618412453756 [ 2972.162496] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.173502] LustreError: 110872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2972.520822] Lustre: server umount lustre-MDT0001 complete [ 2983.604264] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 2986.717689] Key type lgssc unregistered [ 2987.084533] LNet: 114553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2987.098048] LNetError: 114553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2987.122597] LNet: Removed LNI 192.168.204.107@tcp [ 2988.078787] Key type .llcrypt unregistered [ 2988.084636] Key type ._llcrypt unregistered [ 3007.765958] Key type ._llcrypt registered [ 3007.772681] Key type .llcrypt registered [ 3007.962646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3009.921136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3010.123551] alg: No test for adler32 (adler32-zlib) [ 3011.192635] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 3011.489070] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3013.271333] Key type lgssc registered [ 3015.130727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3025.047368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3025.073133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3026.747547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.388597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3036.129137] LustreError: 115779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.147780] LustreError: 115779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3041.262540] LustreError: 115778:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.707472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.095599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3047.008807] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3055.330669] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3062.158946] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3069.867836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.259791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3071.281993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3076.895199] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3084.861289] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3097.059073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.063727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3097.073300] Lustre: Skipped 1 previous similar message [ 3097.075052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3097.094134] Lustre: Skipped 1 previous similar message [ 3102.202383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3102.211662] Lustre: Skipped 1 previous similar message [ 3102.542880] Lustre: server umount lustre-OST0000 complete [ 3107.297286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3107.308029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3107.311806] Lustre: Skipped 1 previous similar message [ 3107.329061] Lustre: Skipped 1 previous similar message [ 3112.429140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3112.439438] Lustre: Skipped 1 previous similar message [ 3112.591583] Lustre: server umount lustre-MDT0000 complete [ 3116.489367] LustreError: 115760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223318 with bad export cookie 16644334312046466328 [ 3116.495345] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3116.514028] LustreError: 115760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3117.135041] Lustre: server umount lustre-MDT0001 complete [ 3128.212378] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3130.899104] Key type lgssc unregistered [ 3131.241385] LNet: 119313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3131.248702] LNetError: 119313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3131.269093] LNet: Removed LNI 192.168.204.107@tcp [ 3132.101665] Key type .llcrypt unregistered [ 3132.103596] Key type ._llcrypt unregistered [ 3156.526421] Key type ._llcrypt registered [ 3156.530642] Key type .llcrypt registered [ 3156.701706] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3157.749253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3157.806580] alg: No test for adler32 (adler32-zlib) [ 3158.921436] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 3159.224378] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3160.895150] Key type lgssc registered [ 3162.185738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3170.113741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3170.156385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.744248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.977851] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3182.052344] LustreError: 120729:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.076179] LustreError: 120729:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3186.281483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.896734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3192.489500] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3199.683566] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3205.940591] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3213.096627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.412862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.525673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3220.385819] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3228.421982] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3238.445169] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3242.373689] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3242.378243] Lustre: Skipped 2 previous similar messages [ 3244.152919] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3244.195374] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3247.066959] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3247.073716] Lustre: Skipped 2 previous similar messages [ 3251.772932] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3251.782610] Lustre: Skipped 1 previous similar message [ 3257.149757] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3257.159343] Lustre: Skipped 2 previous similar messages [ 3259.007057] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.7@tcp 192.168.0.[2,10]@tcp [ 3264.134139] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.7/24@tcp in log lustre-MDT0000 [ 3264.141178] Lustre: Skipped 2 previous similar messages [ 3269.183453] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3269.189056] Lustre: Skipped 1 previous similar message [ 3275.761921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3275.763081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.774547] LustreError: Skipped 2 previous similar messages [ 3275.794129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.306515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3278.310178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3278.326210] LustreError: Skipped 2 previous similar messages [ 3278.328484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.836075] Lustre: server umount lustre-OST0000 complete [ 3288.549505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.554199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3288.566414] Lustre: Skipped 1 previous similar message [ 3288.571804] Lustre: Skipped 1 previous similar message [ 3291.008993] Lustre: server umount lustre-MDT0000 complete [ 3293.664748] LustreError: 120726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.681877] LustreError: 120726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3294.114098] LustreError: 120709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223496 with bad export cookie 15143501405445150840 [ 3294.117228] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3294.133465] LustreError: 120709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3294.440521] Lustre: server umount lustre-MDT0001 complete [ 3304.793783] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3308.053724] Key type lgssc unregistered [ 3308.410652] LNet: 125577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.417822] LNetError: 125577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.439924] LNet: Removed LNI 192.168.204.107@tcp [ 3309.296784] Key type .llcrypt unregistered [ 3309.299669] Key type ._llcrypt unregistered [ 3337.630330] Key type ._llcrypt registered [ 3337.637340] Key type .llcrypt registered [ 3337.782697] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3339.054734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3339.326703] alg: No test for adler32 (adler32-zlib) [ 3340.527905] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 3340.863492] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3342.575134] Key type lgssc registered [ 3343.901156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3352.762547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3352.797520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.422596] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3354.457190] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.7/24@tcp [ 3354.458441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.561060] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3364.842505] LustreError: 127145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3364.891368] LustreError: 127145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3367.963903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.183451] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3368.209365] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.7/24@tcp [ 3368.260922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3373.559203] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3381.647659] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3387.769943] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3395.814777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3396.288899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.364330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3403.554778] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3411.363609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3424.230817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3424.244358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3424.263913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3424.264376] Lustre: Skipped 1 previous similar message [ 3424.279282] Lustre: Skipped 1 previous similar message [ 3428.281958] Lustre: server umount lustre-OST0000 complete [ 3434.468384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3434.474156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3434.480437] Lustre: Skipped 1 previous similar message [ 3434.498361] Lustre: Skipped 1 previous similar message [ 3438.262815] Lustre: server umount lustre-MDT0000 complete [ 3439.587268] LustreError: 127141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.619876] LustreError: 127141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3441.757656] LustreError: 127127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223643 with bad export cookie 13981937442014581669 [ 3441.769140] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3442.060167] Lustre: server umount lustre-MDT0001 complete [ 3451.196874] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3453.491278] Key type lgssc unregistered [ 3453.722665] LNet: 130733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3453.728850] LNetError: 130733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3453.742433] LNet: Removed LNI 192.168.204.107@tcp [ 3454.541515] Key type .llcrypt unregistered [ 3454.544524] Key type ._llcrypt unregistered [ 3476.068861] Key type ._llcrypt registered [ 3476.070519] Key type .llcrypt registered [ 3476.176244] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3491.632239] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3493.277093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3493.324145] alg: No test for adler32 (adler32-zlib) [ 3494.431290] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 3494.712572] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3496.408855] Key type lgssc registered [ 3497.618826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3534.088366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3534.107235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.379669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3535.406091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3535.486340] Lustre: lustre-MDT0000: new disk, initializing [ 3535.582988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.605453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3539.399466] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3548.831837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3548.924428] Lustre: 134134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3548.970861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3548.975445] Lustre: Skipped 1 previous similar message [ 3549.035845] Lustre: lustre-MDT0001: new disk, initializing [ 3549.120572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3549.167080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3549.177606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.326520] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3562.466851] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3568.119404] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3573.848745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.209160] Lustre: lustre-OST0000: new disk, initializing [ 3574.221356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3574.229485] Lustre: 136099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3574.341234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.186549] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3583.069287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3583.078224] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3583.182959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3589.784932] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3594.065238] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3594.262785] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3598.499518] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3598.696338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3603.940129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.948621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.951784] LustreError: Skipped 1 previous similar message [ 3603.953054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.960397] Lustre: Skipped 1 previous similar message [ 3603.978870] Lustre: Skipped 1 previous similar message [ 3606.834138] Lustre: server umount lustre-OST0000 complete [ 3610.595149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3610.604801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.617195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3614.179146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3614.179771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3616.443194] Lustre: server umount lustre-MDT0000 complete [ 3619.847054] LustreError: 134124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223821 with bad export cookie 16863963631813884595 [ 3619.853447] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3619.862430] LustreError: 134124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3620.169911] Lustre: server umount lustre-MDT0001 complete [ 3627.290416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.704064] LustreError: 138320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3627.726268] LustreError: 138320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3627.784981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3632.180593] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3633.126615] LustreError: 138321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3637.218350] LustreError: 138320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3640.640778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.947200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.650296] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3650.554836] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3655.798157] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3661.623395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3661.943393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.635811] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3674.331749] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3679.395232] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3679.696325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3684.741986] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3684.994736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3693.859087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3694.024864] Lustre: lustre-OST0001: new disk, initializing [ 3694.033633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3694.039947] Lustre: 141691:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3694.129579] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3699.837817] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3701.330304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3701.344483] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3701.386399] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3706.863623] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3707.047990] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3711.035996] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3711.227765] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3715.900602] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4fe007b800.ost_server_uuid 50 [ 3717.173525] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4fe007b800.ost_server_uuid in FULL state after 0 sec [ 3727.337520] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3727.340696] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.352095] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3727.352106] Lustre: Skipped 1 previous similar message [ 3727.358529] LustreError: Skipped 1 previous similar message [ 3727.397174] Lustre: Skipped 1 previous similar message [ 3728.154056] Lustre: server umount lustre-OST0001 complete [ 3732.448988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3732.451071] LustreError: 140143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.453882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3732.453895] Lustre: Skipped 1 previous similar message [ 3732.476571] Lustre: Skipped 1 previous similar message [ 3732.514283] LustreError: 140143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3737.576992] LustreError: 140141:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.614356] LustreError: 140141:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3742.694034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3742.702057] Lustre: Skipped 4 previous similar messages [ 3746.271867] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3746.528762] Lustre: server umount lustre-OST0000 complete [ 3752.928411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.938093] Lustre: Skipped 1 previous similar message [ 3756.458235] Lustre: server umount lustre-MDT0000 complete [ 3758.060873] LustreError: 138320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3758.075114] LustreError: 138320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3760.039333] LustreError: 139056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785223961 with bad export cookie 16863963631813885477 [ 3760.043869] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3760.052263] LustreError: 139056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3760.289466] Lustre: server umount lustre-MDT0001 complete [ 3778.396614] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 3780.871742] Key type lgssc unregistered [ 3781.101098] LNet: 144703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3781.117632] LNetError: 144703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3781.131906] LNet: Removed LNI 192.168.204.107@tcp [ 3781.865396] Key type .llcrypt unregistered [ 3781.869321] Key type ._llcrypt unregistered [ 3816.444570] Key type ._llcrypt registered [ 3816.453432] Key type .llcrypt registered [ 3816.566886] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 3830.273604] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 3831.134668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3831.150350] alg: No test for adler32 (adler32-zlib) [ 3832.148740] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 3832.357354] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 3833.999761] Key type lgssc registered [ 3834.786225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3865.525714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3865.542834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3866.704940] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3866.742318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3866.832524] Lustre: lustre-MDT0000: new disk, initializing [ 3866.913387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.935459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3870.668524] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3879.761044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3879.846352] Lustre: 148856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3879.905085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3879.909144] Lustre: Skipped 1 previous similar message [ 3879.974276] Lustre: lustre-MDT0001: new disk, initializing [ 3880.056193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3880.092889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3880.103864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3883.351420] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3892.158676] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3897.036843] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3902.198989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.378552] Lustre: lustre-OST0000: new disk, initializing [ 3902.381697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3902.393691] Lustre: 150820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3902.457328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3906.930823] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3912.220145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3912.230807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3912.319143] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3914.274371] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3917.985686] Lustre: Failing over lustre-OST0000 [ 3918.094307] Lustre: server umount lustre-OST0000 complete [ 3935.168664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3935.419227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.437630] Lustre: Skipped 1 previous similar message [ 3935.460785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3935.476092] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3935.697297] Lustre: 147081:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224121/real 1785224121] req@00000000e3a31265 x1871943076706816/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1785224137 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3936.453626] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3938.271653] Lustre: 147080:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224124/real 1785224124] req@000000004e2eb30f x1871943076707840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785224140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3940.883444] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 3940.884015] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3941.503418] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3943.969057] Lustre: 147081:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224129/real 1785224129] req@000000001979f9d8 x1871943076709120/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785224145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3944.001912] Lustre: 147081:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3944.937336] Lustre: Failing over lustre-MDT0000 [ 3945.219175] Lustre: server umount lustre-MDT0000 complete [ 3945.960351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.984610] Lustre: Skipped 2 previous similar messages [ 3949.025714] Lustre: 147081:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224134/real 1785224134] req@0000000083da8245 x1871943076710144/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785224150 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3949.047756] Lustre: 147081:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3951.074085] LustreError: 149738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.089218] LustreError: 149738:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3951.765108] LustreError: 148862:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.193313] LustreError: 148867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3956.208932] LustreError: 148867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3961.247117] Lustre: 147082:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785224147/real 1785224147] req@000000008ad73c2c x1871943076718080/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785224163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3961.276282] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3961.298685] LustreError: 148866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3961.316874] LustreError: 148866:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3962.710326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3966.437855] LustreError: 148867:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3966.458931] LustreError: 148867:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3971.552478] LustreError: 147078:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001979f9d8 x1871943076722560/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3971.726406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3971.776350] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3972.234619] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3975.038913] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 3977.202436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3977.208563] Lustre: Skipped 1 previous similar message [ 3977.239596] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3977.275888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3982.306451] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.312397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3982.321458] Lustre: Skipped 1 previous similar message [ 3982.329681] Lustre: Skipped 1 previous similar message [ 3984.264186] Lustre: server umount lustre-OST0000 complete [ 3992.551112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.555467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3992.572878] Lustre: Skipped 1 previous similar message [ 3992.592046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3993.707379] Lustre: server umount lustre-MDT0000 complete [ 3996.790876] LustreError: 148846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224198 with bad export cookie 5693721123550573233 [ 3996.796448] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.800941] LustreError: 148846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3997.106486] Lustre: server umount lustre-MDT0001 complete [ 4006.104589] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4008.580732] Key type lgssc unregistered [ 4008.796089] LNet: 154479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.805657] LNetError: 154479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4008.824651] LNet: Removed LNI 192.168.204.107@tcp [ 4009.555581] Key type .llcrypt unregistered [ 4009.559086] Key type ._llcrypt unregistered [ 4039.491837] Key type ._llcrypt registered [ 4039.493905] Key type .llcrypt registered [ 4039.562453] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4040.304111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4040.316816] alg: No test for adler32 (adler32-zlib) [ 4041.289789] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 4041.485769] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4043.175978] Key type lgssc registered [ 4044.082331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4049.806517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4049.820059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.080170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4054.794671] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4060.643855] LustreError: 155998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4060.650985] LustreError: 155998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4062.141722] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4065.055954] LustreError: 156027:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4065.062574] LustreError: 156027:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4065.068374] LustreError: 156027:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 4065.280553] Lustre: server umount lustre-MDT0000 complete [ 4076.549147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4084.774697] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4115.742802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4116.021912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4116.076041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4116.158820] Lustre: lustre-MDT0000: new disk, initializing [ 4116.243897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.264637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4119.887835] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4128.581711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4128.667995] Lustre: 159332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4128.731864] LustreError: 159344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.749822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4128.755403] Lustre: Skipped 1 previous similar message [ 4128.819155] Lustre: lustre-MDT0001: new disk, initializing [ 4128.883573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.914222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4128.920634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4132.237259] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4139.203246] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.231837] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4148.268019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4148.456614] Lustre: lustre-OST0000: new disk, initializing [ 4148.460394] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4148.466824] Lustre: 161297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4148.542858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.968200] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4154.943576] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4154.956859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4155.038911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4160.058496] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4163.078619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4164.291489] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:39:24 (1785224364) [ 4338.657053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4338.665344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4338.678835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.682423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.682986] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.690984] Lustre: Skipped 1 previous similar message [ 4339.697228] Lustre: Skipped 1 previous similar message [ 4344.570216] Lustre: server umount lustre-MDT0000 complete [ 4344.799915] LustreError: 159340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.809444] LustreError: 159340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4346.398169] LustreError: 159325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224548 with bad export cookie 18187175494817774378 [ 4346.400847] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4346.403442] LustreError: 159325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4349.919611] LustreError: 162015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4349.919781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4349.935212] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4351.967926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4352.599388] Lustre: server umount lustre-MDT0001 complete [ 4369.375129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4369.477371] Lustre: server umount lustre-OST0000 complete [ 4374.903977] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4380.187795] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4401.715341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.880951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4401.916744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4401.970398] Lustre: lustre-MDT0000: new disk, initializing [ 4402.030082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.044754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4404.686898] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4412.158681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4412.225684] Lustre: 165944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4412.270924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4412.273457] Lustre: Skipped 1 previous similar message [ 4412.316509] Lustre: lustre-MDT0001: new disk, initializing [ 4412.385097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4412.420827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4412.432972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4415.290959] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4421.544674] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4425.584902] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4429.946357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4430.093831] Lustre: lustre-OST0000: new disk, initializing [ 4430.099334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4430.106393] Lustre: 167912:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4430.160470] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.900458] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4438.550534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4438.556294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4438.603955] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4439.816664] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4442.308565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4442.448696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.398791] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4445.560445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4448.740137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4448.740506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.746786] LustreError: Skipped 1 previous similar message [ 4448.747695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4448.757330] Lustre: Skipped 1 previous similar message [ 4452.988826] Lustre: server umount lustre-OST0000 complete [ 4458.466305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4458.470876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.478856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4458.481975] Lustre: Skipped 1 previous similar message [ 4461.305132] Lustre: server umount lustre-MDT0000 complete [ 4463.197575] LustreError: 167910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224665 with bad export cookie 18187175494817933208 [ 4463.205957] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4463.370947] Lustre: server umount lustre-MDT0001 complete [ 4467.864894] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:44:28 (1785224668) [ 4476.233883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4481.403518] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4500.355496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.486844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4500.506211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4500.550796] Lustre: lustre-MDT0000: new disk, initializing [ 4500.607916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4500.623137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4502.827545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4508.922353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.980607] Lustre: 172659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4509.137863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4511.087202] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4516.616963] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4520.150557] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4524.167634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.323123] Lustre: 174639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4527.338479] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4532.233596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4532.302917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4532.814791] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4537.317915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.319509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4537.330925] Lustre: Skipped 1 previous similar message [ 4537.334924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4537.335546] LustreError: Skipped 1 previous similar message [ 4537.337928] Lustre: Skipped 1 previous similar message [ 4538.859325] Lustre: server umount lustre-OST0000 complete [ 4542.435763] LustreError: 172667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.447164] LustreError: 172667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4542.978676] Lustre: server umount lustre-MDT0000 complete [ 4544.916231] LustreError: 174638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224746 with bad export cookie 18187175494817934090 [ 4544.919175] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4544.924285] LustreError: 174638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4545.078092] Lustre: server umount lustre-MDT0001 complete [ 4549.340356] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:45:49 (1785224749) [ 4557.203713] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 4561.844348] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4580.771871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.900499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4580.919165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4580.921313] Lustre: Skipped 3 previous similar messages [ 4580.977572] Lustre: lustre-MDT0000: new disk, initializing [ 4580.981395] Lustre: Skipped 2 previous similar messages [ 4581.033366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.041684] Lustre: Skipped 2 previous similar messages [ 4581.057466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4581.063590] Lustre: Skipped 2 previous similar messages [ 4583.404159] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4590.057916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4590.146450] Lustre: 178947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4590.331581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4592.821467] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4598.806116] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4601.814972] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4605.812941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.055458] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4606.057231] Lustre: 180928:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4606.062349] Lustre: Skipped 1 previous similar message [ 4609.781618] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4611.601577] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4615.785807] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4621.795563] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4621.795735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4621.800692] LustreError: Skipped 1 previous similar message [ 4621.806633] Lustre: Skipped 4 previous similar messages [ 4625.449864] Lustre: server umount lustre-OST0000 complete [ 4631.520380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4633.643185] Lustre: server umount lustre-MDT0000 complete [ 4635.432109] LustreError: 178940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224837 with bad export cookie 18187175494817935161 [ 4635.434254] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4635.437957] LustreError: 178940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4635.596380] Lustre: server umount lustre-MDT0001 complete [ 4640.628104] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4642.231328] Key type lgssc unregistered [ 4642.404668] LNet: 182779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4642.409815] LNetError: 182779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4642.419554] LNet: Removed LNI 192.168.204.107@tcp [ 4642.923606] Key type .llcrypt unregistered [ 4642.925689] Key type ._llcrypt unregistered [ 4656.837080] Key type ._llcrypt registered [ 4656.838605] Key type .llcrypt registered [ 4656.896487] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4657.475918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4657.490059] alg: No test for adler32 (adler32-zlib) [ 4658.400143] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 4658.532755] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4660.143159] Key type lgssc registered [ 4660.814441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.062116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.078100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.300108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4668.478679] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4673.257720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4673.492633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4675.811372] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4679.946332] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4683.311332] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4687.257628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4687.451235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4690.350814] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4691.568953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4694.744465] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4719.596165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4719.600386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4719.607989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4719.616910] Lustre: Skipped 1 previous similar message [ 4720.444252] Lustre: server umount lustre-OST0000 complete [ 4729.838949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4729.840271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4729.850948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4729.850958] Lustre: Skipped 1 previous similar message [ 4729.898520] Lustre: Skipped 1 previous similar message [ 4733.273541] Lustre: server umount lustre-MDT0000 complete [ 4734.944328] LustreError: 185563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4734.972854] LustreError: 185563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4740.047311] LustreError: 186013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785224941 with bad export cookie 17581911183098820480 [ 4740.048879] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4740.071845] LustreError: 186013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4740.908420] Lustre: server umount lustre-MDT0001 complete [ 4757.857564] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 4761.480346] Key type lgssc unregistered [ 4761.902784] LNet: 187745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4761.913793] LNetError: 187745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4761.941582] LNet: Removed LNI 192.168.204.107@tcp [ 4763.151112] Key type .llcrypt unregistered [ 4763.153436] Key type ._llcrypt unregistered [ 4808.461984] Key type ._llcrypt registered [ 4808.463719] Key type .llcrypt registered [ 4808.543501] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 4810.746915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4810.821079] alg: No test for adler32 (adler32-zlib) [ 4811.915304] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 4812.383760] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 4814.143239] Key type lgssc registered [ 4815.579713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4825.047584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4825.079052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4826.606283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.508798] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4836.845158] LustreError: 189165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4836.866311] LustreError: 189165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4841.916466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.970363] LustreError: 189166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.368964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4846.755723] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4853.939445] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4860.613884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4868.719228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.069772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4872.123436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4875.683921] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4885.642712] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4897.764160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4897.765320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4897.795501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4897.808751] Lustre: Skipped 1 previous similar message [ 4901.082885] Lustre: server umount lustre-OST0000 complete [ 4909.280354] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e4fc8145800.ost_server_uuid 50 [ 4910.943902] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4927.457243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.488248] Lustre: Skipped 1 previous similar message [ 4927.500023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.516585] Lustre: Skipped 2 previous similar messages [ 4931.121440] Lustre: server umount lustre-MDT0000 complete [ 4932.584167] LustreError: 189162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.600081] LustreError: 189162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4934.603845] LustreError: 189146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785225136 with bad export cookie 6446851310698034128 [ 4934.625426] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.251604] Lustre: server umount lustre-MDT0001 complete [ 4943.727174] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:52:23 (1785225143) [ 4950.334473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.792209] LustreError: 193086:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.869374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.870375] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4956.136572] LustreError: 193087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4964.647824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4965.062515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4970.018216] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 4978.406545] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4984.505119] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4990.699688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4991.058821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4995.204235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4998.170166] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5005.418575] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5012.473389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5012.779935] Lustre: lustre-OST0001: new disk, initializing [ 5012.785058] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5012.805122] Lustre: 195929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5012.930265] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5018.296100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5018.306296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5018.456044] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5020.678125] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5030.837729] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.962613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.966717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5033.994331] Lustre: Skipped 2 previous similar messages [ 5034.007265] Lustre: Skipped 1 previous similar message [ 5036.725503] Lustre: lustre-OST0000: Not available for connect from 192.168.204.7@tcp (stopping) [ 5039.950399] Lustre: server umount lustre-OST0000 complete [ 5041.811441] LustreError: 194909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.855919] LustreError: 194909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5047.387917] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5047.636231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5052.978885] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5053.253812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5059.556015] LustreError: 194909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5059.588748] LustreError: 194909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5069.794116] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5069.799329] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5069.801423] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5069.801431] Lustre: Skipped 2 previous similar messages [ 5069.807550] Lustre: Skipped 1 previous similar message [ 5069.815907] LustreError: Skipped 1 previous similar message [ 5073.876801] Lustre: server umount lustre-OST0001 complete [ 5080.035696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5080.042255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5080.052853] Lustre: Skipped 1 previous similar message [ 5080.060060] Lustre: Skipped 2 previous similar messages [ 5084.653993] Lustre: server umount lustre-MDT0000 complete [ 5088.647783] LustreError: 195928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785225290 with bad export cookie 6446851310698035199 [ 5088.649649] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5088.664355] LustreError: 195928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5089.024752] Lustre: server umount lustre-MDT0001 complete [ 5115.837516] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:55:15 (1785225315) [ 5123.235958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.531329] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5123.557208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5123.817903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5129.457681] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5139.621832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.762257] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5139.843440] Lustre: 199369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5146.191450] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5155.907286] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5162.522554] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.582094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.859249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5178.565738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5179.480807] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5188.281425] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5196.127282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.225228] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5196.470357] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5196.477889] Lustre: Skipped 2 previous similar messages [ 5202.447645] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5210.358612] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5214.186600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5214.206484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.240682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5219.544832] Lustre: server umount lustre-OST0000 complete [ 5220.325076] LustreError: 202131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.344233] LustreError: 202131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5235.171398] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5235.179591] LustreError: Skipped 1 previous similar message [ 5235.181490] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5235.192861] Lustre: Skipped 1 previous similar message [ 5241.051702] Lustre: server umount lustre-OST0001 complete [ 5251.041179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5251.050229] Lustre: Skipped 11 previous similar messages [ 5251.472646] Lustre: server umount lustre-MDT0000 complete [ 5255.482768] LustreError: 199360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785225457 with bad export cookie 6446851310698036606 [ 5255.485085] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5255.493729] LustreError: 199360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5255.948303] Lustre: server umount lustre-MDT0001 complete [ 5281.314762] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:58:01 (1785225481) [ 5300.883762] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 5311.415613] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 5349.107148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.349498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5349.368820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5349.447457] Lustre: lustre-MDT0000: new disk, initializing [ 5349.494992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5349.514730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5354.085710] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5364.915489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.031283] Lustre: 207471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5365.093606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5365.097581] Lustre: Skipped 1 previous similar message [ 5365.167696] Lustre: lustre-MDT0001: new disk, initializing [ 5365.305091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5365.316471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5369.901907] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5381.069904] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5388.557314] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5395.586878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5395.884739] Lustre: lustre-OST0000: new disk, initializing [ 5395.895585] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5395.913616] Lustre: 209440:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5401.673301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5401.693762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5401.807469] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5402.181911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5411.413491] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5416.575247] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5416.807156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5421.734872] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5421.928781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5427.683162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.684789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.684886] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5427.684890] LustreError: Skipped 1 previous similar message [ 5427.707662] Lustre: Skipped 4 previous similar messages [ 5427.733494] Lustre: Skipped 1 previous similar message [ 5429.943777] Lustre: server umount lustre-OST0000 complete [ 5436.910651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5440.367851] Lustre: server umount lustre-MDT0000 complete [ 5443.041270] LustreError: 207484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.053830] LustreError: 207484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 5444.495498] LustreError: 208276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785225646 with bad export cookie 6446851310698038167 [ 5444.500588] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5444.508323] LustreError: 208276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5445.068616] Lustre: server umount lustre-MDT0001 complete [ 5453.315740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.997590] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5466.805552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.049578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5480.390235] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5489.819570] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5498.079902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5504.758965] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5513.514662] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5518.269199] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5518.484560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5523.044621] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5523.242787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5524.962932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5524.969976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.980473] Lustre: Skipped 2 previous similar messages [ 5539.295962] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5539.527186] Lustre: server umount lustre-OST0000 complete [ 5546.707993] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5552.296228] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5580.932879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5588.894948] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5592.614597] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.622943] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5592.627054] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5592.677224] Lustre: Skipped 1 previous similar message [ 5596.656265] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5599.917878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5607.928620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5607.930148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5607.949161] LustreError: Skipped 1 previous similar message [ 5607.950369] Lustre: Skipped 10 previous similar messages [ 5611.159388] Lustre: server umount lustre-OST0000 complete [ 5622.560389] Lustre: server umount lustre-MDT0000 complete [ 5623.265539] LustreError: 211652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.302192] LustreError: 211652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5627.115781] LustreError: 211637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785225829 with bad export cookie 6446851310698039049 [ 5627.122670] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.716634] Lustre: server umount lustre-MDT0001 complete [ 5637.006195] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:03:56 (1785225836) [ 5644.427155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.107564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5645.118419] Lustre: Skipped 6 previous similar messages [ 5650.759422] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5658.919835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5663.072226] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5670.360610] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5676.349203] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5682.846348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5689.489034] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5697.047524] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.126779] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5701.369735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5705.438901] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5705.620794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5711.985696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.210359] Lustre: lustre-OST0001: new disk, initializing [ 5712.219862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5712.227325] Lustre: 221005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5719.310325] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5721.672467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5721.684158] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5721.777411] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5728.803937] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5733.299848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5733.524461] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5738.101793] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5738.338182] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5742.561603] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.572863] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.589199] Lustre: Skipped 5 previous similar messages [ 5754.335451] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5754.480678] Lustre: server umount lustre-OST0001 complete [ 5761.538559] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5761.760269] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5767.282818] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5767.520399] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5797.765740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5803.508908] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5803.539357] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5803.542981] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5805.049230] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5813.926512] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5815.852121] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5824.104254] Lustre: server umount lustre-OST0001 complete [ 5847.719900] Lustre: server umount lustre-OST0000 complete [ 5858.296847] Lustre: server umount lustre-MDT0000 complete [ 5862.578719] LustreError: 219479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785226064 with bad export cookie 6446851310698040008 [ 5862.590156] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.591174] LustreError: 219479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5869.252580] Lustre: server umount lustre-MDT0001 complete [ 5896.955540] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:08:16 (1785226096) [ 5904.777635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5905.044310] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5905.064518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5910.702412] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5920.549161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.651331] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5920.688692] Lustre: 226252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5926.188879] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5935.423051] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.316753] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5951.619624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.029220] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5960.255822] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5969.186153] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5981.377136] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5981.489813] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5989.890385] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 5998.657238] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6000.580913] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6005.754826] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6005.972202] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6010.738696] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6010.957975] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6016.625654] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e4fd0127000.ost_server_uuid 50 [ 6018.141940] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e4fd0127000.ost_server_uuid in IDLE state after 0 sec [ 6019.298852] Lustre: Permanently deactivating lustre-OST0001 [ 6019.327428] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6024.107584] Lustre: Permanently reactivating lustre-OST0001 [ 6024.142853] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6024.149087] Lustre: Skipped 2 previous similar messages [ 6029.798535] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6029.814312] LustreError: Skipped 6 previous similar messages [ 6029.822664] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6029.860680] Lustre: Skipped 7 previous similar messages [ 6029.876402] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6029.884448] Lustre: Skipped 19 previous similar messages [ 6034.981990] Lustre: server umount lustre-OST0001 complete [ 6037.495381] LustreError: 228096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6037.519770] LustreError: 228096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 36 previous similar messages [ 6054.373113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6054.558550] Lustre: server umount lustre-OST0000 complete [ 6065.133963] Lustre: server umount lustre-MDT0000 complete [ 6068.743747] LustreError: 226993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785226270 with bad export cookie 6446851310698041303 [ 6068.748582] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.758920] LustreError: 226993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6069.158397] Lustre: server umount lustre-MDT0001 complete [ 6096.103378] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:11:36 (1785226296) [ 6104.058543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6104.370666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6104.421364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6104.429791] Lustre: Skipped 2 previous similar messages [ 6109.286916] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6117.765305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.898957] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6117.934450] Lustre: 232952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6122.756657] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6130.307900] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6137.051875] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6144.400102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6144.609296] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6144.637978] Lustre: Permanently deactivating lustre-OST0000 [ 6144.660337] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6151.732058] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6152.074689] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6160.713990] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6167.998025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6168.165420] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6168.430209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6168.437385] Lustre: Skipped 11 previous similar messages [ 6175.413163] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6176.950567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6183.415294] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6185.588993] Lustre: Permanently reactivating lustre-OST0000 [ 6185.619926] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6185.632442] Lustre: Skipped 2 previous similar messages [ 6186.883467] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6186.899497] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 000000009f777c13, cur 1785226388 deadline 1785226383 last 1785226353 [ 6186.990794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6207.224583] Lustre: server umount lustre-OST0001 complete [ 6217.887384] Lustre: server umount lustre-OST0000 complete [ 6236.128857] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6236.324972] Lustre: server umount lustre-MDT0000 complete [ 6240.115445] LustreError: 232943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785226442 with bad export cookie 6446851310698042969 [ 6240.137437] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6240.443781] Lustre: server umount lustre-MDT0001 complete [ 6251.843271] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6255.110380] Key type lgssc unregistered [ 6255.495620] LNet: 237830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.502904] LNetError: 237830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6255.521276] LNet: Removed LNI 192.168.204.107@tcp [ 6256.546360] Key type .llcrypt unregistered [ 6256.547826] Key type ._llcrypt unregistered [ 6278.292072] Key type ._llcrypt registered [ 6278.297215] Key type .llcrypt registered [ 6278.433233] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6279.534685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6279.551419] alg: No test for adler32 (adler32-zlib) [ 6280.525851] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 6280.830571] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6282.535233] Key type lgssc registered [ 6283.733215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6294.481292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6294.515267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6296.117452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6300.457453] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6305.259505] LustreError: 239279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.285314] LustreError: 239279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6309.181406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6309.263689] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6309.312667] Lustre: 239265:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6309.337523] Lustre: Permanently deactivating lustre-MDT0001 [ 6309.367475] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6309.678950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.085400] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6312.098113] LustreError: 239309:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6312.118875] LustreError: 239309:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6312.134363] LustreError: 239309:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6314.232499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6321.665788] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6327.975373] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6335.212626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6335.652335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6340.780692] LustreError: 241116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6340.820350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6342.088546] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6346.213681] LustreError: 241116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.881453] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6351.340990] LustreError: 241119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.447305] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6356.217378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.514266] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.601838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6361.615672] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6362.217985] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6371.194118] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6375.436767] Lustre: Permanently reactivating lustre-MDT0001 [ 6375.451462] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6375.456222] Lustre: Skipped 2 previous similar messages [ 6377.403163] Lustre: lustre-MDT0001: Client 67eb5ca2-71fd-4e4b-9448-3e36a6ca27a9 (at 192.168.204.7@tcp) reconnecting [ 6384.266403] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6386.348830] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6392.913564] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6393.133380] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6398.129672] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6398.471254] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6401.812070] Lustre: Permanently deactivating lustre-MDT0001 [ 6401.834545] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6401.840651] Lustre: Skipped 2 previous similar messages [ 6409.340949] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6414.819376] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.829638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6414.837018] Lustre: Skipped 1 previous similar message [ 6414.867091] Lustre: Skipped 3 previous similar messages [ 6417.029320] Lustre: server umount lustre-MDT0000 complete [ 6419.939788] LustreError: 242993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.962506] LustreError: 242993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6421.564661] LustreError: 239259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785226623 with bad export cookie 1305767324557504974 [ 6421.570917] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6421.575556] LustreError: 239259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6421.910438] Lustre: server umount lustre-MDT0001 complete [ 6428.517476] Lustre: server umount lustre-OST0000 complete [ 6442.697527] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:17:22 (1785226642) [ 6455.781864] Lustre: server umount lustre-OST0001 complete [ 6464.300233] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 6476.057986] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6516.074066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6516.281756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6516.290981] Lustre: Skipped 2 previous similar messages [ 6516.333843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6516.428803] Lustre: lustre-MDT0000: new disk, initializing [ 6516.527719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.595852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6521.433398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6533.332585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.536692] Lustre: 247539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6533.620760] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6533.627507] Lustre: Skipped 1 previous similar message [ 6533.773918] Lustre: lustre-MDT0001: new disk, initializing [ 6533.905235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6533.944315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6533.960738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6538.783194] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6551.067804] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6557.988301] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6566.315249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6566.837489] Lustre: lustre-OST0000: new disk, initializing [ 6566.840746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6566.846878] Lustre: 249509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6566.981627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.771609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6574.802492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6575.110618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6577.814252] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6589.829684] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6592.770635] LustreError: 247546:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6600.925361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6601.070772] Lustre: lustre-OST0001: new disk, initializing [ 6601.075971] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6601.081328] Lustre: 250624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6601.178267] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6602.874390] LustreError: 247546:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6602.902375] LustreError: 247545:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6607.181957] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6608.970928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6608.978859] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6609.031409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6612.944204] LustreError: 247545:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6615.989686] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.485957] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6619.618341] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6619.620112] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6619.627603] LustreError: Skipped 1 previous similar message [ 6619.628524] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6619.650480] Lustre: Skipped 3 previous similar messages [ 6620.850114] Lustre: lustre-OST0001: Not available for connect from 192.168.204.7@tcp (stopping) [ 6620.859083] Lustre: Skipped 1 previous similar message [ 6624.743569] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6625.227887] Lustre: server umount lustre-OST0001 complete [ 6625.935717] LustreError: 249494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6641.637425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6641.642650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6641.648552] LustreError: Skipped 1 previous similar message [ 6641.650847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6641.650855] Lustre: Skipped 1 previous similar message [ 6641.700289] Lustre: Skipped 1 previous similar message [ 6645.222687] LustreError: 249492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6645.243543] LustreError: 249492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6647.332980] Lustre: server umount lustre-OST0000 complete [ 6651.873451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6651.880807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6651.892057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6651.894188] Lustre: Skipped 3 previous similar messages [ 6657.928135] Lustre: server umount lustre-MDT0000 complete [ 6661.734144] LustreError: 247533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785226863 with bad export cookie 1305767324557507788 [ 6661.745335] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6661.748478] LustreError: 247533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6668.212156] Lustre: server umount lustre-MDT0001 complete [ 6680.658770] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6683.542682] Key type lgssc unregistered [ 6683.901805] LNet: 252658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6683.918477] LNetError: 252658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6683.941214] LNet: Removed LNI 192.168.204.107@tcp [ 6685.031779] Key type .llcrypt unregistered [ 6685.034120] Key type ._llcrypt unregistered [ 6723.683461] Key type ._llcrypt registered [ 6723.686652] Key type .llcrypt registered [ 6723.826597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6725.224657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6725.284229] alg: No test for adler32 (adler32-zlib) [ 6726.421464] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 6726.789970] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6728.487822] Key type lgssc registered [ 6729.845780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6737.896733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6737.931299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.201746] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6739.236076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6739.501791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6744.367863] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6753.263309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6753.392323] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6753.415995] Lustre: 254868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6753.675374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6757.634239] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6765.419931] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.677661] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.418276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.635847] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6777.819860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.478876] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6785.666661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6789.848884] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.826646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6841.837650] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.858425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.343200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6842.363374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6842.379796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.390924] Lustre: Skipped 1 previous similar message [ 6847.133173] Lustre: server umount lustre-OST0000 complete [ 6850.889602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6863.256591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6863.561817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6867.683154] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6867.702035] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6869.431499] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 6876.410180] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.057226] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6883.068587] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6919.147024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6919.148561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6919.174196] LustreError: Skipped 1 previous similar message [ 6919.174914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6919.207654] Lustre: Skipped 1 previous similar message [ 6921.734728] Lustre: server umount lustre-OST0000 complete [ 6927.840810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6927.850368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6927.868373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6927.876173] Lustre: Skipped 1 previous similar message [ 6931.670064] Lustre: server umount lustre-MDT0000 complete [ 6935.546888] LustreError: 254859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227137 with bad export cookie 935076758845867389 [ 6935.549422] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6935.553754] LustreError: 254859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6935.895081] Lustre: server umount lustre-MDT0001 complete [ 6947.810110] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 6950.129882] Key type lgssc unregistered [ 6950.405796] LNet: 260286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6950.414906] LNetError: 260286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6950.430681] LNet: Removed LNI 192.168.204.107@tcp [ 6951.153199] Key type .llcrypt unregistered [ 6951.160074] Key type ._llcrypt unregistered [ 6975.891641] Key type ._llcrypt registered [ 6975.894744] Key type .llcrypt registered [ 6976.029926] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 6976.854850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6977.010406] alg: No test for adler32 (adler32-zlib) [ 6978.051963] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 6978.275326] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 6980.023200] Key type lgssc registered [ 6981.227526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6989.252549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6989.276248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.844530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6995.386944] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7001.059383] LustreError: 261704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7001.102658] LustreError: 261704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7004.597625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.077077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7011.352557] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7019.720382] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7025.563278] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7031.811976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.118544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7035.192086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7038.581442] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7045.876772] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7071.207386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7071.216185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.231049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7075.296409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7075.303921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7075.311498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7075.339596] Lustre: Skipped 1 previous similar message [ 7076.764691] Lustre: server umount lustre-OST0000 complete [ 7081.964741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7081.977710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7081.992492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7085.543969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7085.544375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7086.988579] Lustre: server umount lustre-MDT0000 complete [ 7090.656459] LustreError: 261704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7090.687730] LustreError: 261704:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7091.005455] LustreError: 261685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227292 with bad export cookie 15724525353107343690 [ 7091.017197] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7091.032967] LustreError: 261685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7091.414864] Lustre: server umount lustre-MDT0001 complete [ 7103.725218] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7107.837728] Key type lgssc unregistered [ 7108.234187] LNet: 266009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7108.242423] LNetError: 266009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7108.266281] LNet: Removed LNI 192.168.204.107@tcp [ 7108.993501] Key type .llcrypt unregistered [ 7108.997879] Key type ._llcrypt unregistered [ 7128.998975] Key type ._llcrypt registered [ 7129.005182] Key type .llcrypt registered [ 7129.171208] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7130.213540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7130.342131] alg: No test for adler32 (adler32-zlib) [ 7131.414927] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7131.718876] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7133.367565] Key type lgssc registered [ 7134.633317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7143.457365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7143.504419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.197308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.774366] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7155.683520] LustreError: 267242:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7155.699476] LustreError: 267242:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7158.671346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.155386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7164.021744] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7171.569536] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7177.729713] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7186.099616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.515068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.639229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7193.403508] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7201.790188] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7216.630629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7216.644050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.665887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.690441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7219.694365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7219.706307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.753065] Lustre: Skipped 1 previous similar message [ 7222.691687] Lustre: server umount lustre-OST0000 complete [ 7229.925937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.938402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7233.549387] Lustre: server umount lustre-MDT0000 complete [ 7235.044713] LustreError: 267239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7235.060487] LustreError: 267239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7237.693608] LustreError: 267223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227439 with bad export cookie 6629902613389959626 [ 7237.705962] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7237.710238] LustreError: 267223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7238.393661] Lustre: server umount lustre-MDT0001 complete [ 7250.484330] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7253.450461] Key type lgssc unregistered [ 7253.843903] LNet: 270923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7253.854636] LNetError: 270923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7253.867687] LNet: Removed LNI 192.168.204.107@tcp [ 7255.001953] Key type .llcrypt unregistered [ 7255.006338] Key type ._llcrypt unregistered [ 7280.897214] Key type ._llcrypt registered [ 7280.904119] Key type .llcrypt registered [ 7281.031401] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7282.341936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7282.465603] alg: No test for adler32 (adler32-zlib) [ 7283.633762] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7283.898537] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7285.591199] Key type lgssc registered [ 7286.819734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7294.921926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7294.951036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.492756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.502420] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7306.729653] LustreError: 272340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.754253] LustreError: 272340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7310.049880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.477117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7315.151281] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7323.215382] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7329.548439] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.683280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7338.126194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7341.247333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7345.488457] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7354.170586] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7381.472851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7381.483214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7381.493222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7381.508899] Lustre: Skipped 1 previous similar message [ 7381.520243] Lustre: Skipped 1 previous similar message [ 7385.287598] Lustre: server umount lustre-OST0000 complete [ 7391.713637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7391.724166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7391.724419] Lustre: Skipped 1 previous similar message [ 7395.152883] Lustre: server umount lustre-MDT0000 complete [ 7396.838644] LustreError: 272333:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.873508] LustreError: 272333:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7399.715397] LustreError: 272319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227601 with bad export cookie 206415493399291244 [ 7399.731361] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7399.732443] LustreError: 272319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7400.143932] Lustre: server umount lustre-MDT0001 complete [ 7410.935065] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7413.520513] Key type lgssc unregistered [ 7413.817561] LNet: 276647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7413.824136] LNetError: 276647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7413.847474] LNet: Removed LNI 192.168.204.107@tcp [ 7414.625142] Key type .llcrypt unregistered [ 7414.627993] Key type ._llcrypt unregistered [ 7431.590653] Key type ._llcrypt registered [ 7431.593894] Key type .llcrypt registered [ 7431.729383] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7432.794685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7433.023786] alg: No test for adler32 (adler32-zlib) [ 7434.038808] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7434.268952] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7435.983623] Key type lgssc registered [ 7437.222827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7444.894643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7444.908857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.419213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7450.334671] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7456.738480] LustreError: 277883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7456.781327] LustreError: 277883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7459.756116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.270591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7465.307647] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7474.424675] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7480.693211] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7487.558038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.940278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.094525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7494.893984] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7502.596579] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7517.664873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7517.684312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7517.701070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7519.714430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7519.721263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7519.729085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7519.753670] Lustre: Skipped 1 previous similar message [ 7523.562461] Lustre: server umount lustre-OST0000 complete [ 7529.961577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.968372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7529.984700] Lustre: Skipped 1 previous similar message [ 7534.418932] Lustre: server umount lustre-MDT0000 complete [ 7535.086311] LustreError: 277891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.173027] LustreError: 277882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.210414] LustreError: 277882:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7539.484121] LustreError: 277848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227741 with bad export cookie 13485429731548294025 [ 7539.508906] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7539.516058] LustreError: 277848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7540.136535] Lustre: server umount lustre-MDT0001 complete [ 7551.831430] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7554.928214] Key type lgssc unregistered [ 7555.199799] LNet: 281589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7555.206590] LNetError: 281589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7555.221242] LNet: Removed LNI 192.168.204.107@tcp [ 7556.037764] Key type .llcrypt unregistered [ 7556.045455] Key type ._llcrypt unregistered [ 7567.871722] Key type ._llcrypt registered [ 7567.877027] Key type .llcrypt registered [ 7568.023771] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7585.571784] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7602.934662] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7603.761103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7603.957325] alg: No test for adler32 (adler32-zlib) [ 7604.984265] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7605.282745] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7607.063140] Key type lgssc registered [ 7608.191908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7646.537645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7646.559956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7647.850291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7647.879563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7647.940557] Lustre: lustre-MDT0000: new disk, initializing [ 7647.998501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.011229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7651.997265] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7662.902484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.075714] Lustre: 285141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7663.147972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7663.151984] Lustre: Skipped 1 previous similar message [ 7663.233427] Lustre: lustre-MDT0001: new disk, initializing [ 7663.316584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7663.382413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7663.394440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7667.595895] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7677.504957] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7685.142306] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7693.129791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7693.502557] Lustre: lustre-OST0000: new disk, initializing [ 7693.506214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7693.514171] Lustre: 287110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7693.611924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7701.581817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7701.590190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7701.699320] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7701.777799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7712.590858] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7719.549079] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7720.043640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7725.531389] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7725.759616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7732.711080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7732.713197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.722285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7732.743147] Lustre: Skipped 1 previous similar message [ 7732.758530] Lustre: Skipped 1 previous similar message [ 7734.117811] Lustre: server umount lustre-OST0000 complete [ 7740.389939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7740.409597] LustreError: Skipped 1 previous similar message [ 7740.423659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7740.468744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.945079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7742.951093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.544125] Lustre: server umount lustre-MDT0000 complete [ 7750.251932] LustreError: 285133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785227952 with bad export cookie 9410450738423859048 [ 7750.267099] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7750.268049] LustreError: 285133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7750.761765] Lustre: server umount lustre-MDT0001 complete [ 7760.148833] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:39:19 (1785227959) [ 7767.722714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.509664] LustreError: 289523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.528617] LustreError: 289523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7768.659218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.665814] LustreError: 289524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.385965] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7778.787432] LustreError: 289523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7783.926255] LustreError: 289524:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.253925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.952616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7792.972246] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7801.372929] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7807.875888] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7816.203849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.604548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7824.792492] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 7835.408661] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7853.537446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7853.553978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7853.571961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7853.581328] Lustre: Skipped 1 previous similar message [ 7858.402411] Lustre: server umount lustre-OST0000 complete [ 7864.809627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7864.827854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.828338] LustreError: Skipped 1 previous similar message [ 7864.829678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7864.829684] Lustre: Skipped 2 previous similar messages [ 7864.854652] Lustre: Skipped 2 previous similar messages [ 7869.188501] Lustre: server umount lustre-MDT0000 complete [ 7869.923015] LustreError: 291185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.938486] LustreError: 291185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7872.912319] LustreError: 289503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228074 with bad export cookie 9410450738423859937 [ 7872.914218] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7872.926663] LustreError: 289503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7873.590993] Lustre: server umount lustre-MDT0001 complete [ 7888.609425] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 7892.679337] Key type lgssc unregistered [ 7892.989423] LNet: 293084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7892.995720] LNetError: 293084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.061424] LNet: Removed LNI 192.168.204.107@tcp [ 7895.377523] Key type .llcrypt unregistered [ 7895.379593] Key type ._llcrypt unregistered [ 7924.510546] Key type ._llcrypt registered [ 7924.513270] Key type .llcrypt registered [ 7924.655517] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 7941.747219] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 7943.120972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7943.266163] alg: No test for adler32 (adler32-zlib) [ 7944.340355] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 7944.588231] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 7946.319159] Key type lgssc registered [ 7947.560093] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7998.192428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7998.248285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.548718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7999.581948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7999.713861] Lustre: lustre-MDT0000: new disk, initializing [ 7999.823847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7999.848466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8005.532049] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8017.487564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8017.582946] Lustre: 296911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8017.626509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8017.637917] Lustre: Skipped 1 previous similar message [ 8017.721486] Lustre: lustre-MDT0001: new disk, initializing [ 8017.833558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8017.891201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8017.904344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8023.045746] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8033.232568] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8040.188414] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8047.499622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8047.784390] Lustre: lustre-OST2710: new disk, initializing [ 8047.791621] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8047.809814] Lustre: 298880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8047.967209] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8054.547985] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8056.410903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8056.429831] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8056.534342] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8064.118983] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8071.512080] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.661560] Lustre: lustre-OST03e8: new disk, initializing [ 8071.668168] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8071.673532] Lustre: 299944:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8071.777201] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8077.979852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8077.990798] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8078.207162] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8078.950218] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8087.839341] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8095.811967] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8096.098976] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8100.082315] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8100.264272] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8105.583072] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8105.900518] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8110.807548] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8110.977732] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8115.175908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8115.182697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8115.196754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8117.383632] Lustre: server umount lustre-MDT0000 complete [ 8119.265394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.284841] Lustre: Skipped 1 previous similar message [ 8122.170535] LustreError: 296903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228324 with bad export cookie 127296041811090478 [ 8122.182637] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8122.187895] LustreError: 296903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8122.604881] Lustre: server umount lustre-MDT0001 complete [ 8137.101500] Lustre: server umount lustre-OST2710 complete [ 8151.064839] Lustre: server umount lustre-OST03e8 complete [ 8170.906183] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 8179.405374] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8211.481798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.663631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8211.695654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8211.753478] Lustre: lustre-MDT0000: new disk, initializing [ 8211.810930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8211.822156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8214.992141] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8229.870449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.008261] Lustre: 305199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8230.080380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8230.089265] Lustre: Skipped 1 previous similar message [ 8230.167866] Lustre: lustre-MDT0001: new disk, initializing [ 8230.246640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.281321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8230.288066] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8236.016525] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8247.128581] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8254.354217] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8261.799459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.040196] Lustre: lustre-OST0000: new disk, initializing [ 8262.043665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8262.051091] Lustre: 307166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8262.202474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.976402] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8269.416727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8269.427053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8269.608592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8280.550337] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8285.223971] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8285.503954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8290.496486] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8290.814944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8295.403230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8295.404340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.413431] LustreError: Skipped 1 previous similar message [ 8295.417597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8295.446802] Lustre: Skipped 2 previous similar messages [ 8299.448509] Lustre: server umount lustre-OST0000 complete [ 8305.633408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8305.640479] Lustre: Skipped 1 previous similar message [ 8305.641865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.648540] Lustre: Skipped 2 previous similar messages [ 8309.791970] Lustre: server umount lustre-MDT0000 complete [ 8310.755240] LustreError: 305210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.786201] LustreError: 305210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8314.354540] LustreError: 305192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228516 with bad export cookie 127296041811092158 [ 8314.360424] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8314.366266] LustreError: 305192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8314.676494] Lustre: server umount lustre-MDT0001 complete [ 8326.261411] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:48:45 (1785228525) [ 8328.300858] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8330.824876] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:48:50 (1785228530) [ 8335.896696] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8365.817709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8366.271560] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8366.306839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8366.521780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.801981] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8381.306438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8381.386278] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8381.411526] Lustre: 311025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8387.243066] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8410.093603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8410.272622] Lustre: Denying initial registration attempt from nid 192.168.204.107@tcp, specified as failover [ 8410.276856] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8410.286778] LustreError: 312478:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8410.291634] LustreError: 312478:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8410.296127] LustreError: 312478:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8410.335672] Lustre: server umount lustre-OST0000 complete [ 8410.340025] LustreError: 312478:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8417.760952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8417.771952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8417.788411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8422.579390] Lustre: server umount lustre-MDT0000 complete [ 8424.420564] LustreError: 311032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8424.438909] LustreError: 311032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8425.940279] LustreError: 311016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228627 with bad export cookie 127296041811093040 [ 8425.946769] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8426.303103] Lustre: server umount lustre-MDT0001 complete [ 8436.818629] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8439.574910] Key type lgssc unregistered [ 8439.930634] LNet: 313604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8439.948416] LNetError: 313604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8439.971271] LNet: Removed LNI 192.168.204.107@tcp [ 8440.857118] Key type .llcrypt unregistered [ 8440.859511] Key type ._llcrypt unregistered [ 8451.797280] Key type ._llcrypt registered [ 8451.799289] Key type .llcrypt registered [ 8451.879674] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8452.632316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8452.662465] alg: No test for adler32 (adler32-zlib) [ 8453.624240] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 8453.776435] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8455.407194] Key type lgssc registered [ 8456.144598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8497.157881] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8506.626809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8506.648153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.914517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8507.930931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8508.079921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8512.033501] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8517.120670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.184424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8517.201305] Lustre: 316367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8517.338953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8519.323068] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8524.958656] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8530.170736] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8547.696257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8547.949431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8548.132435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8553.962360] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8560.997947] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8563.179742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8563.183255] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.184767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8563.209265] Lustre: Skipped 1 previous similar message [ 8568.310423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8568.323927] Lustre: Skipped 2 previous similar messages [ 8569.496699] Lustre: server umount lustre-OST0000 complete [ 8573.921621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8573.928085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.942253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8578.529269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8578.529829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8580.444599] Lustre: server umount lustre-MDT0000 complete [ 8586.649208] LustreError: 316360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228788 with bad export cookie 7089803954436948522 [ 8586.664718] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8586.665154] LustreError: 316360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8587.065509] Lustre: server umount lustre-MDT0001 complete [ 8598.808572] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8601.076857] Key type lgssc unregistered [ 8601.360702] LNet: 319944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8601.374234] LNetError: 319944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8602.409739] LNet: Removed LNI 192.168.204.107@tcp [ 8603.540440] Key type .llcrypt unregistered [ 8603.544161] Key type ._llcrypt unregistered [ 8631.153565] Key type ._llcrypt registered [ 8631.156874] Key type .llcrypt registered [ 8631.346629] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8632.496616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8632.536345] alg: No test for adler32 (adler32-zlib) [ 8633.684968] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 8633.973624] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8635.728575] Key type lgssc registered [ 8636.971645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8644.260820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8644.284628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.603584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8649.051805] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8655.845114] LustreError: 321364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.862161] LustreError: 321364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8657.388338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.834101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8662.582261] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8671.602677] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8678.319702] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8687.994368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8688.579608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.269811] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8705.295537] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8718.307455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.312097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.321125] Lustre: Skipped 2 previous similar messages [ 8718.338352] Lustre: Skipped 2 previous similar messages [ 8721.493339] Lustre: server umount lustre-MDT0000 complete [ 8723.434440] LustreError: 321357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8723.459742] LustreError: 321357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8725.510060] LustreError: 321343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785228927 with bad export cookie 12722017798739548553 [ 8725.514273] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8725.522329] LustreError: 321343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8728.545630] LustreError: 321357:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.547077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.557445] LustreError: 321357:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8728.590487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8728.595020] Lustre: Skipped 1 previous similar message [ 8730.594631] LustreError: 322368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8730.595705] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8730.614655] LustreError: 322368:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8730.629130] Lustre: Skipped 1 previous similar message [ 8732.030702] Lustre: server umount lustre-MDT0001 complete [ 8738.442892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8747.877763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.073089] LustreError: 320673:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000504f02ff x1871948110923520/t0(0) o250->MGC192.168.204.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8751.560088] LustreError: 323195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.660207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.738575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8756.101543] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8760.802395] LustreError: 324956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8760.816717] LustreError: 324956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8761.832828] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8764.531643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.960965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8769.491656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8770.037430] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8776.526964] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8778.257541] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8784.398332] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8786.189654] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8792.548531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8792.554568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.575479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8795.617857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.639400] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8797.608695] Lustre: server umount lustre-OST0000 complete [ 8805.857807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8805.864058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.886568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.897674] Lustre: Skipped 3 previous similar messages [ 8808.343348] Lustre: server umount lustre-MDT0000 complete [ 8810.977389] LustreError: 324960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8810.990781] LustreError: 324960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8812.505887] LustreError: 321345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229014 with bad export cookie 12722017798739555700 [ 8812.528068] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.797360] Lustre: server umount lustre-MDT0001 complete [ 8824.437326] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 8827.495327] Key type lgssc unregistered [ 8827.837762] LNet: 327629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8827.843139] LNetError: 327629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8827.867693] LNet: Removed LNI 192.168.204.107@tcp [ 8828.897485] Key type .llcrypt unregistered [ 8828.899972] Key type ._llcrypt unregistered [ 8852.852908] Key type ._llcrypt registered [ 8852.855588] Key type .llcrypt registered [ 8852.953504] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 8854.031774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8854.135205] alg: No test for adler32 (adler32-zlib) [ 8855.106527] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 8855.345550] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 8857.023365] Key type lgssc registered [ 8858.248618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8866.988874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8867.005314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.499122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.948074] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8879.090305] LustreError: 329046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8879.110644] LustreError: 329046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8881.808611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8882.112527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.908185] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8895.298403] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8902.260236] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8906.977699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8906.981244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8907.003477] Lustre: Skipped 1 previous similar message [ 8907.015124] Lustre: Skipped 1 previous similar message [ 8912.356067] LustreError: 330222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8912.379302] LustreError: 330222:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8912.445863] Lustre: server umount lustre-MDT0000 complete [ 8916.681787] LustreError: 329027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229118 with bad export cookie 9366768829316988581 [ 8916.691114] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8916.699238] LustreError: 329027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8917.303370] Lustre: server umount lustre-MDT0001 complete [ 8934.340694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8934.667972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8934.702200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8934.938828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.864755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8949.854369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8950.016983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8950.049664] Lustre: 331633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8950.385797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.561077] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 8965.178306] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8971.782396] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8983.042321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8983.237817] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8983.254683] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8983.261313] LustreError: 333520:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 8983.268261] LustreError: 333520:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8983.276846] LustreError: 333520:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8983.362201] Lustre: server umount lustre-OST0000 complete [ 8983.364837] LustreError: 333520:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8995.642818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8995.931560] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8996.344295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9002.542512] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9003.090703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9010.245875] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9018.776962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.875360] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9018.937948] Lustre: lustre-OST0001: new disk, initializing [ 9018.943241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9018.948636] Lustre: 335061:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9019.054282] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9027.556738] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9028.790276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9028.809889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9028.956237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9037.951525] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9044.449548] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9044.462982] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.498606] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9044.511162] Lustre: Skipped 1 previous similar message [ 9047.931984] Lustre: server umount lustre-OST0001 complete [ 9049.587797] LustreError: 333987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9049.607852] LustreError: 333987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9054.178154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9054.193275] LustreError: Skipped 1 previous similar message [ 9054.196891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.205440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9054.232340] Lustre: Skipped 2 previous similar messages [ 9054.251110] Lustre: Skipped 1 previous similar message [ 9054.699865] LustreError: 333986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9054.735663] LustreError: 333986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9059.819808] LustreError: 333987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.358430] Lustre: server umount lustre-OST0000 complete [ 9068.005136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9068.022167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.051210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.795680] Lustre: server umount lustre-MDT0000 complete [ 9075.169179] LustreError: 333089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9075.190509] LustreError: 333089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9077.689680] LustreError: 331627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229279 with bad export cookie 9366768829316989134 [ 9077.702328] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9077.703854] LustreError: 331627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9078.080974] Lustre: server umount lustre-MDT0001 complete [ 9093.647883] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9096.913405] Key type lgssc unregistered [ 9097.224122] LNet: 337033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9097.240590] LNetError: 337033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9097.281859] LNet: Removed LNI 192.168.204.107@tcp [ 9098.258864] Key type .llcrypt unregistered [ 9098.262704] Key type ._llcrypt unregistered [ 9180.575699] Key type ._llcrypt registered [ 9180.578225] Key type .llcrypt registered [ 9180.664834] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [ 9201.204616] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9202.609584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9202.691087] alg: No test for adler32 (adler32-zlib) [ 9203.771925] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 9204.057751] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9205.783188] Key type lgssc registered [ 9207.289197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9247.894891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9247.931382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.183212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.216464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.350375] Lustre: lustre-MDT0000: new disk, initializing [ 9249.436958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.452729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9254.493616] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9265.255316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.510955] Lustre: 341970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9265.623219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9265.627121] Lustre: Skipped 1 previous similar message [ 9265.824134] Lustre: lustre-MDT0001: new disk, initializing [ 9266.079181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9266.149946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9266.172804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9272.019267] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9281.049988] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9285.380857] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9290.004649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9290.191440] Lustre: lustre-OST0000: new disk, initializing [ 9290.194994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9290.200600] Lustre: 343938:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9290.274938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9293.733486] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9298.484026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9298.489718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9298.548738] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9299.266965] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9301.632255] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9301.740908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9304.219550] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9304.331860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9308.652378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.672342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9308.685596] Lustre: Skipped 1 previous similar message [ 9308.708378] Lustre: Skipped 1 previous similar message [ 9311.939704] Lustre: server umount lustre-OST0000 complete [ 9317.344619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9317.353476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.388787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9318.884135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9318.885578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.402261] Lustre: server umount lustre-MDT0000 complete [ 9326.632234] LustreError: 341962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229528 with bad export cookie 1425704824231985375 [ 9326.633866] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9326.641108] LustreError: 341962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9327.056884] Lustre: server umount lustre-MDT0001 complete [ 9336.188478] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:05:36 (1785229536) [ 9344.831398] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:05:44 (1785229544) [ 9355.046380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9355.605088] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.627311] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9355.720706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9360.761869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9360.865315] LustreError: 346691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9365.986265] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.065651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.511666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.517963] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9384.719859] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9392.406633] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9399.521572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.845681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9406.652981] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9414.221820] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9422.712272] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9425.471173] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9427.629472] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9429.978704] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9432.336118] Lustre: Failing over lustre-MDT0000 [ 9432.572377] Lustre: server umount lustre-MDT0000 complete [ 9435.618900] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9435.623347] LustreError: 346685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9435.633566] Lustre: Skipped 1 previous similar message [ 9440.758335] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9440.787052] LustreError: 346690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9450.981821] LustreError: 346687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9451.013241] LustreError: 346687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9452.000526] Lustre: 340194:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785229637/real 1785229637] req@000000002a7cbd8a x1871948708645760/t0(0) o400->MGC192.168.204.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785229653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9452.036444] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9452.187381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.223827] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x13c91efbe2e88887 [ 9461.241587] Lustre: MGC192.168.204.107@tcp: Connection restored to 0@lo (at 0@lo) [ 9461.440782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.552079] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9465.811772] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9466.003362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9466.864042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9466.898809] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9466.955298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9475.888557] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9478.467579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9484.031731] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9487.356154] Lustre: Failing over lustre-MDT0000 [ 9488.004433] Lustre: server umount lustre-MDT0000 complete [ 9491.642053] LustreError: 347829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9491.658782] LustreError: 347829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 9492.453874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.459510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9492.488946] Lustre: Skipped 3 previous similar messages [ 9507.294384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.370710] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9507.652061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.696832] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9508.013624] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9512.333916] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9512.951372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9512.965500] Lustre: Skipped 2 previous similar messages [ 9513.018356] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9513.053769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9526.247400] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9528.772682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9538.529688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9538.530493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.533493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9538.533504] Lustre: Skipped 1 previous similar message [ 9538.553601] Lustre: Skipped 1 previous similar message [ 9543.651242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9543.667782] Lustre: Skipped 4 previous similar messages [ 9543.853840] Lustre: server umount lustre-MDT0000 complete [ 9548.145868] LustreError: 346673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229750 with bad export cookie 1425704824231988084 [ 9548.162865] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9548.583124] Lustre: server umount lustre-MDT0001 complete [ 9563.270630] Lustre: server umount lustre-OST0000 complete [ 9579.915189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9580.260768] LustreError: 353032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.283129] LustreError: 353032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 9580.317380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9584.733658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9594.655456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9601.308606] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9610.465321] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9617.642663] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9625.737687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9627.010502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9632.559646] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9640.659834] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9650.660352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9650.672407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9650.693655] Lustre: Skipped 1 previous similar message [ 9650.698342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.710464] Lustre: Skipped 1 previous similar message [ 9654.705963] Lustre: server umount lustre-OST0000 complete [ 9666.896729] Lustre: server umount lustre-MDT0000 complete [ 9670.625730] LustreError: 353032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9670.659033] LustreError: 353032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9672.288705] LustreError: 353014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785229874 with bad export cookie 1425704824231988826 [ 9672.292736] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9672.296120] LustreError: 353014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9672.551794] Lustre: server umount lustre-MDT0001 complete [ 9686.755903] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [ 9690.363333] Key type lgssc unregistered [ 9690.595939] LNet: 356567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9690.608330] LNetError: 356567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9690.631981] LNet: Removed LNI 192.168.204.107@tcp [ 9691.550530] Key type .llcrypt unregistered [ 9691.554673] Key type ._llcrypt unregistered [ 9724.748484] Key type ._llcrypt registered [ 9724.750617] Key type .llcrypt registered [ 9724.863228] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [ 9725.467250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9725.559163] alg: No test for adler32 (adler32-zlib) [ 9726.667462] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [ 9727.001795] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [ 9728.759409] Key type lgssc registered [ 9730.619394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9740.391720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9740.404225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9741.922012] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.586528] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9752.035308] LustreError: 358084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9752.064846] LustreError: 358084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9757.157422] LustreError: 358085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.844817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.274338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9763.517369] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9772.275795] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9779.365251] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9788.765244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9789.017682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9798.344973] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9808.104958] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9811.643945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9824.741446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9824.751554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9824.761209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.301406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9827.303657] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9827.311487] Lustre: Skipped 1 previous similar message [ 9827.349785] Lustre: Skipped 1 previous similar message [ 9830.982503] Lustre: server umount lustre-MDT0000 complete [ 9832.418255] LustreError: 358085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9832.437272] LustreError: 358085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9835.052138] LustreError: 358064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230036 with bad export cookie 16673761828036902017 [ 9835.065962] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.066902] LustreError: 358064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9835.481277] Lustre: server umount lustre-MDT0001 complete [ 9849.955282] Lustre: server umount lustre-OST0000 complete [ 9865.098423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9865.455917] LustreError: 361818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9865.525673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9870.538774] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9870.827215] LustreError: 361819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.369977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.745689] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9885.445901] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9892.718254] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9899.498281] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9907.817198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9908.250995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9910.285077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9916.470169] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [ 9925.443404] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9940.973724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.982385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9941.012073] Lustre: Skipped 2 previous similar messages [ 9941.027352] Lustre: Skipped 3 previous similar messages [ 9945.514539] Lustre: server umount lustre-MDT0000 complete [ 9946.080557] LustreError: 363228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9946.105367] LustreError: 363228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9950.169243] LustreError: 361800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230152 with bad export cookie 16673761828036903130 [ 9950.196788] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.210038] LustreError: 361800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9950.952499] Lustre: server umount lustre-MDT0001 complete [ 9968.397908] Lustre: server umount lustre-OST0000 complete [ 9987.032295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9987.574357] LustreError: 365735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9987.645779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9992.578087] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10001.932126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10008.442439] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10020.529340] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10027.666581] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10035.450596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.744322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10035.750637] Lustre: Skipped 1 previous similar message [10042.094234] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10051.316697] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10058.350178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10068.449443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10068.461114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.488590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10074.083433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10074.096036] Lustre: Skipped 1 previous similar message [10074.351905] Lustre: server umount lustre-OST0000 complete [10079.203214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10079.204048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.214128] LustreError: Skipped 1 previous similar message [10079.254908] Lustre: Skipped 2 previous similar messages [10084.321263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10084.331965] Lustre: Skipped 4 previous similar messages [10084.551060] Lustre: server umount lustre-MDT0000 complete [10089.050876] LustreError: 365716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230290 with bad export cookie 16673761828036904145 [10089.052417] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10089.064863] LustreError: 365716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10089.562530] Lustre: server umount lustre-MDT0001 complete [10103.023481] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10106.371545] Key type lgssc unregistered [10106.692518] LNet: 369289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10106.701299] LNetError: 369289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10107.789838] LNet: Removed LNI 192.168.204.107@tcp [10109.074348] Key type .llcrypt unregistered [10109.076413] Key type ._llcrypt unregistered [10137.644854] Key type ._llcrypt registered [10137.650278] Key type .llcrypt registered [10137.777848] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10139.876185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10139.923206] alg: No test for adler32 (adler32-zlib) [10141.093154] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [10141.402936] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10143.127911] Key type lgssc registered [10144.358759] Lustre: Echo OBD driver; http://www.lustre.org/ [10153.921511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10153.950745] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.970600] LustreError: 370734:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10153.986300] LustreError: 370734:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10153.992383] LustreError: 370734:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10153.997805] LustreError: 370734:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10154.005449] LustreError: 370734:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10163.275044] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10163.287179] LustreError: 371131:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10163.322801] LustreError: 371131:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10163.332717] LustreError: 371131:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10163.347435] LustreError: 371131:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10163.359519] LustreError: 371131:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10185.849692] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10188.896794] Key type lgssc unregistered [10189.204991] LNet: 372135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10189.214311] LNetError: 372135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10189.235862] LNet: Removed LNI 192.168.204.107@tcp [10190.144598] Key type .llcrypt unregistered [10190.149833] Key type ._llcrypt unregistered [10209.504348] Key type ._llcrypt registered [10209.506709] Key type .llcrypt registered [10209.643990] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [10229.026380] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10230.481748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10230.686433] alg: No test for adler32 (adler32-zlib) [10231.790638] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [10232.104468] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10233.784043] Key type lgssc registered [10234.938514] Lustre: Echo OBD driver; http://www.lustre.org/ [10275.510530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10275.538193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.788740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10276.822027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10276.950740] Lustre: lustre-MDT0000: new disk, initializing [10277.087071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.125709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10282.767162] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10293.421295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.588240] Lustre: 375294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10293.768988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10293.776800] Lustre: Skipped 1 previous similar message [10293.893294] Lustre: lustre-MDT0001: new disk, initializing [10293.985134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10294.030484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10294.041870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10298.710368] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10311.650848] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10319.747895] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10328.355105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10328.600386] Lustre: lustre-OST0000: new disk, initializing [10328.604523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10328.612721] Lustre: 377264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10328.694582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10336.584127] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10336.891570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10336.912427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10337.054522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10347.607847] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10352.112936] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10352.311230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10356.863705] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10357.121377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10362.857118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10362.860030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10362.868752] LustreError: Skipped 1 previous similar message [10362.874366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10362.899806] Lustre: Skipped 1 previous similar message [10365.136656] Lustre: server umount lustre-OST0000 complete [10371.047486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10371.065227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10371.102029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10371.110407] Lustre: Skipped 1 previous similar message [10373.088287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10373.096831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10376.754126] Lustre: server umount lustre-MDT0000 complete [10380.684912] LustreError: 375286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230582 with bad export cookie 12663603016480880543 [10380.708747] LustreError: 375286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10380.710795] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10380.981581] Lustre: server umount lustre-MDT0001 complete [10391.685828] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10393.990803] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:23:13 (1785230593) [10400.721942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.069303] LustreError: 379725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.088664] LustreError: 379725:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10401.130316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10406.374747] LustreError: 379726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.401816] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10410.466553] LustreError: 379725:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.591707] LustreError: 379726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.784487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10416.302579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10421.385976] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10430.586880] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10438.146744] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10444.775978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10445.234378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10452.222792] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10460.377517] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10468.714941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10468.895347] Lustre: lustre-OST0001: new disk, initializing [10468.905793] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10468.913225] Lustre: 382566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10469.001321] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10474.560472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10474.574926] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10474.691685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10475.310444] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10485.432279] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10490.344446] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10490.349097] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.357842] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10490.365857] Lustre: Skipped 1 previous similar message [10490.371796] Lustre: Skipped 1 previous similar message [10495.468160] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10495.482042] Lustre: Skipped 3 previous similar messages [10495.920915] Lustre: server umount lustre-OST0001 complete [10497.674774] LustreError: 381550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.708881] LustreError: 381548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.745323] LustreError: 381548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10515.938747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10515.942448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10515.950311] Lustre: Skipped 1 previous similar message [10515.955673] Lustre: Skipped 1 previous similar message [10526.182935] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.210301] LustreError: 381549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10527.199786] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10527.567984] Lustre: server umount lustre-OST0000 complete [10533.859855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10533.871432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10533.883146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10533.887223] Lustre: Skipped 4 previous similar messages [10537.953297] Lustre: server umount lustre-MDT0000 complete [10542.667258] LustreError: 379707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230744 with bad export cookie 12663603016480881425 [10542.677375] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10542.684021] LustreError: 379707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10549.462797] Lustre: server umount lustre-MDT0001 complete [10564.074958] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10566.496563] Key type lgssc unregistered [10566.716718] LNet: 384555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10566.720745] LNetError: 384555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10566.734253] LNet: Removed LNI 192.168.204.107@tcp [10567.512348] Key type .llcrypt unregistered [10567.514663] Key type ._llcrypt unregistered [10625.356472] Key type ._llcrypt registered [10625.361619] Key type .llcrypt registered [10625.608480] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [10627.460583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10627.656601] alg: No test for adler32 (adler32-zlib) [10629.063775] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [10629.705081] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [10631.535456] Key type lgssc registered [10633.172297] Lustre: Echo OBD driver; http://www.lustre.org/ [10642.636267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10642.663797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10643.886588] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10643.912223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10644.051599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10648.319280] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10656.995526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10657.174411] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10657.235847] Lustre: 387355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10657.423318] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10657.661380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10662.322289] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10670.619714] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10678.937609] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10683.360965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10683.372461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10683.411394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.437418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.447991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10686.456427] Lustre: Skipped 1 previous similar message [10687.795461] Lustre: server umount lustre-MDT0000 complete [10692.341501] LustreError: 387347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785230894 with bad export cookie 1143643144054627772 [10692.352807] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.362250] LustreError: 387347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10692.755198] Lustre: server umount lustre-MDT0001 complete [10704.905987] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:28:24 (1785230904) [10711.515302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10711.869329] LustreError: 389871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.898418] LustreError: 389871:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10711.965805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10716.072058] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10717.153984] LustreError: 389872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.280238] LustreError: 389871:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.175413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.672591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10731.097822] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10738.964301] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10745.131913] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10752.568548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10752.781226] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10752.894873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10758.525407] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10766.482342] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10775.207467] Lustre: Permanently deactivating lustre-OST0000 [10775.230603] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10782.768330] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10784.698767] LustreError: 392858:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10784.710740] LustreError: 392858:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10784.716803] LustreError: 392858:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10787.376777] LustreError: 392906:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10787.383482] LustreError: 392906:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10795.514686] Lustre: server umount lustre-OST0000 complete [10798.339727] LustreError: 393157:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [10798.343816] LustreError: 393157:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10800.102983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10800.119568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10800.120020] Lustre: Skipped 1 previous similar message [10807.265484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10807.270807] Lustre: Skipped 4 previous similar messages [10809.422368] Lustre: server umount lustre-MDT0000 complete [10812.384878] LustreError: 392373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.409859] LustreError: 392373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10813.790561] LustreError: 389853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231015 with bad export cookie 1143643144054628346 [10813.804819] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10813.808911] LustreError: 389853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10823.306122] Lustre: server umount lustre-MDT0001 complete [10831.807047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10836.647804] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10839.482892] LustreError: 394316:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10846.289553] Lustre: 394655:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.107@tcp [10848.842665] Lustre: server umount MGS complete [10861.892646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.349468] LustreError: 395380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.364816] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10862.368813] Lustre: Skipped 1 previous similar message [10862.411344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10867.127202] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10872.813864] LustreError: 395380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10872.849737] LustreError: 395380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10877.137737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.506758] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10882.633242] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10890.609339] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10896.953753] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10899.196325] Lustre: Permanently reactivating lustre-OST0000 [10899.222587] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10899.227682] Lustre: Skipped 2 previous similar messages [10927.583610] LustreError: 397159:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10935.233363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.604450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10935.612516] Lustre: Skipped 1 previous similar message [10942.490904] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [10951.250912] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10958.308729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10958.311853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.317263] LustreError: Skipped 1 previous similar message [10958.320455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10958.324154] Lustre: Skipped 1 previous similar message [10958.334498] Lustre: Skipped 1 previous similar message [10958.828599] Lustre: server umount lustre-OST0000 complete [10963.427689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.448781] Lustre: Skipped 1 previous similar message [10968.837189] Lustre: server umount lustre-MDT0000 complete [10972.396208] LustreError: 396115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231174 with bad export cookie 1143643144054629403 [10972.398306] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10972.406880] LustreError: 396115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10972.750529] Lustre: server umount lustre-MDT0001 complete [10985.899135] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [10988.540540] Key type lgssc unregistered [10988.941088] LNet: 399743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10988.948412] LNetError: 399743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10988.968219] LNet: Removed LNI 192.168.204.107@tcp [10989.828792] Key type .llcrypt unregistered [10989.830362] Key type ._llcrypt unregistered [11010.279543] Key type ._llcrypt registered [11010.283350] Key type .llcrypt registered [11010.406561] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11025.805627] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11026.959438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11027.170640] alg: No test for adler32 (adler32-zlib) [11028.353405] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11028.610170] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11030.287480] Key type lgssc registered [11031.340628] Lustre: Echo OBD driver; http://www.lustre.org/ [11069.318781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11069.366630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.640776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11070.675612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11070.760297] Lustre: lustre-MDT0000: new disk, initializing [11070.836648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11070.854775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11075.267928] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11085.411444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.536598] Lustre: 402900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11085.621298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11085.625645] Lustre: Skipped 1 previous similar message [11085.676052] Lustre: lustre-MDT0001: new disk, initializing [11085.758716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11085.807076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11085.818055] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11090.714931] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11101.889091] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11108.499856] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11115.190421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11115.533939] Lustre: lustre-OST0000: new disk, initializing [11115.537596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11115.544557] Lustre: 404871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11115.670769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11121.926080] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11125.337325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11125.348093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11125.457176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11131.606964] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11136.081551] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11136.290598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11140.603591] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11140.802202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11146.210632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11146.216042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.230985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11146.240495] Lustre: Skipped 1 previous similar message [11146.256316] Lustre: Skipped 1 previous similar message [11149.279888] Lustre: server umount lustre-OST0000 complete [11156.449988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11156.467549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11156.474235] Lustre: Skipped 1 previous similar message [11159.813280] Lustre: server umount lustre-MDT0000 complete [11163.715273] LustreError: 402891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231365 with bad export cookie 2109458549642166836 [11163.720047] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11163.734355] LustreError: 402891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11164.073081] Lustre: server umount lustre-MDT0001 complete [11172.963930] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11174.863930] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:36:14 (1785231374) [11181.641497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.140863] LustreError: 407332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.161760] LustreError: 407332:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11182.264567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11186.917545] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11187.680676] LustreError: 407331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11192.808726] LustreError: 407332:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11195.860050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.247403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11201.010743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11209.420591] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.391984] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11221.693728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11221.931510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11228.692304] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11237.546618] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11239.512447] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11242.335069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11248.611078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11248.619268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11248.619918] LustreError: Skipped 1 previous similar message [11248.621225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11248.646409] Lustre: Skipped 2 previous similar messages [11248.655356] Lustre: Skipped 1 previous similar message [11253.729436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11253.737293] Lustre: Skipped 1 previous similar message [11254.299334] Lustre: server umount lustre-OST0000 complete [11258.849740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11258.866952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11258.883984] Lustre: Skipped 1 previous similar message [11264.970262] Lustre: server umount lustre-MDT0000 complete [11269.094948] LustreError: 407331:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.120219] LustreError: 407331:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11269.210200] LustreError: 407312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231471 with bad export cookie 2109458549642167718 [11269.215293] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11269.226436] LustreError: 407312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11269.694729] Lustre: server umount lustre-MDT0001 complete [11282.795597] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11285.822505] Key type lgssc unregistered [11286.233507] LNet: 411060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11286.237480] LNetError: 411060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11286.276488] LNet: Removed LNI 192.168.204.107@tcp [11287.169601] Key type .llcrypt unregistered [11287.171772] Key type ._llcrypt unregistered [11309.033112] Key type ._llcrypt registered [11309.035861] Key type .llcrypt registered [11309.179683] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11309.474852] Key type .llcrypt unregistered [11309.479120] Key type ._llcrypt unregistered [11327.999388] Key type ._llcrypt registered [11328.002482] Key type .llcrypt registered [11328.098984] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11328.751208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11328.764086] alg: No test for adler32 (adler32-zlib) [11329.880566] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11330.072236] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11331.727233] Key type lgssc registered [11332.739469] Lustre: Echo OBD driver; http://www.lustre.org/ [11340.124946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11340.146405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.662596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11346.633464] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11352.032231] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11352.071487] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11355.506249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11355.912687] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11355.946713] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11355.948766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11361.248252] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11361.266166] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11361.845345] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11366.370761] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.389760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11366.399553] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11369.311895] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11371.491495] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11371.506079] LustreError: 413272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11376.498651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11376.875257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11381.591956] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11391.971707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11391.984325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11391.996734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11392.406600] Lustre: server umount lustre-OST0000 complete [11397.603508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.604922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11397.605652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11397.616380] Lustre: Skipped 1 previous similar message [11402.442521] Lustre: server umount lustre-MDT0000 complete [11402.721436] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.736236] LustreError: 413273:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11406.161814] LustreError: 413253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231608 with bad export cookie 6956821103418060245 [11406.178871] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11416.712987] Lustre: server umount lustre-MDT0001 complete [11427.372310] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11429.884440] Key type lgssc unregistered [11430.143865] LNet: 416419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11430.152954] LNetError: 416419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11430.171639] LNet: Removed LNI 192.168.204.107@tcp [11430.867371] Key type .llcrypt unregistered [11430.870225] Key type ._llcrypt unregistered [11454.554128] Key type ._llcrypt registered [11454.562026] Key type .llcrypt registered [11454.638321] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11455.468720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11455.488560] alg: No test for adler32 (adler32-zlib) [11456.566972] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11456.801499] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11458.495686] Key type lgssc registered [11459.872507] Lustre: Echo OBD driver; http://www.lustre.org/ [11469.350584] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11469.390418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11484.961509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11490.081326] LustreError: 417813:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.107@tcp: failed processing log, type 4: rc = -110 [11516.767243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11522.315904] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11529.908881] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11537.645355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.245467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.306507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11542.986970] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11543.527469] LustreError: 417839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.597130] LustreError: 417839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11548.644740] LustreError: 417838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.144275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11551.446652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11555.421486] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11563.615628] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11569.735757] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11577.321816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11577.328354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11577.348104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11578.346842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.354150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11578.406195] Lustre: Skipped 1 previous similar message [11578.817304] Lustre: server umount lustre-OST0000 complete [11583.462174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11583.469278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11583.472516] Lustre: Skipped 1 previous similar message [11583.491753] Lustre: Skipped 1 previous similar message [11588.576996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.581474] Lustre: Skipped 1 previous similar message [11588.765834] Lustre: server umount lustre-MDT0000 complete [11592.427226] LustreError: 417819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231794 with bad export cookie 2671981197830346938 [11592.428678] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.435225] LustreError: 417819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11592.697799] Lustre: server umount lustre-MDT0001 complete [11603.131301] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11606.193824] Key type lgssc unregistered [11606.555185] LNet: 421380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11606.574965] LNetError: 421380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11606.589304] LNet: Removed LNI 192.168.204.107@tcp [11607.571075] Key type .llcrypt unregistered [11607.576491] Key type ._llcrypt unregistered [11634.032552] Key type ._llcrypt registered [11634.034636] Key type .llcrypt registered [11634.164147] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11635.384702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11635.456892] alg: No test for adler32 (adler32-zlib) [11636.534935] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11636.829419] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11638.511544] Key type lgssc registered [11639.537060] Lustre: Echo OBD driver; http://www.lustre.org/ [11646.934202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11646.952191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.453219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.237814] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11658.738819] LustreError: 422799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11658.760129] LustreError: 422799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11661.706036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.116953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11666.432851] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11675.168548] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11681.416366] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11688.907063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11689.263724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11693.371165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11695.432576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11702.468401] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11708.386109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11708.406232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.430599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11710.641327] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.7@tcp (stopping) [11711.971054] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11711.974426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11711.985665] Lustre: Skipped 1 previous similar message [11712.006322] Lustre: Skipped 2 previous similar messages [11713.708191] Lustre: server umount lustre-MDT0000 complete [11716.080548] LustreError: 422778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.7@tcp arrived at 1785231917 with bad export cookie 5280258417902239351 [11716.093187] LustreError: 422778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11717.089284] LustreError: 422799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11717.115339] LustreError: 422799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11718.555082] LustreError: 423532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785231920 with bad export cookie 5280258417902238462 [11718.561626] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11718.570670] LustreError: 423532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11722.208171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11722.210709] LustreError: 423834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11722.217763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11722.220106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11722.278540] LustreError: 423834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11724.820965] Lustre: server umount lustre-OST0000 complete [11728.357304] LustreError: 422795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11728.377404] LustreError: 422795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11731.167445] Lustre: server umount lustre-MDT0001 complete [11742.507407] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11746.539142] Key type lgssc unregistered [11746.982922] LNet: 426483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11747.001859] LNetError: 426483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11747.035675] LNet: Removed LNI 192.168.204.107@tcp [11748.262230] Key type .llcrypt unregistered [11748.265812] Key type ._llcrypt unregistered [11780.006719] Key type ._llcrypt registered [11780.007835] Key type .llcrypt registered [11780.104513] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11781.224628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11781.404182] alg: No test for adler32 (adler32-zlib) [11782.444699] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11782.733816] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11784.450806] Key type lgssc registered [11785.656132] Lustre: Echo OBD driver; http://www.lustre.org/ [11796.603934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11796.665945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.528195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.185362] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11808.740711] LustreError: 427903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11808.767342] LustreError: 427903:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11812.381082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11812.872879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11818.233494] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11826.962106] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11834.312667] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11842.526356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11842.907127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11843.984525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11851.136970] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [11858.264508] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11862.498583] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11862.505108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11862.523299] Lustre: Skipped 1 previous similar message [11862.536178] Lustre: Skipped 1 previous similar message [11865.247339] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [11867.619812] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11867.631108] Lustre: Skipped 1 previous similar message [11868.270508] Lustre: server umount lustre-MDT0001 complete [11870.356958] LustreError: 430240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.737165] LustreError: 428631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.756179] LustreError: 428631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11874.785268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11874.798971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11874.829063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11877.861919] LustreError: 427904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11880.909621] Lustre: server umount lustre-OST0000 complete [11882.979140] LustreError: 428631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11884.760068] Lustre: server umount lustre-MDT0000 complete [11896.069397] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [11898.555590] Key type lgssc unregistered [11898.796849] LNet: 431590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11898.804890] LNetError: 431590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11898.820846] LNet: Removed LNI 192.168.204.107@tcp [11899.558925] Key type .llcrypt unregistered [11899.562810] Key type ._llcrypt unregistered [11922.187046] Key type ._llcrypt registered [11922.190800] Key type .llcrypt registered [11922.353453] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [11939.609300] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [11940.995567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11941.023800] alg: No test for adler32 (adler32-zlib) [11942.217595] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [11942.553400] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [11944.336115] Key type lgssc registered [11945.978789] Lustre: Echo OBD driver; http://www.lustre.org/ [11991.281890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12006.512695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12006.547475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.835809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12007.890880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12007.967171] Lustre: lustre-MDT0000: new disk, initializing [12008.059426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12008.074278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12011.987700] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12025.908580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12026.157608] Lustre: lustre-OST0000: new disk, initializing [12026.162947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12026.166809] Lustre: Skipped 1 previous similar message [12026.171535] Lustre: 436624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12026.252349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12032.389781] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12035.151662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12035.162324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12035.192657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12041.760528] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12057.884618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12058.007879] Lustre: 435608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12058.069044] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12058.122877] Lustre: lustre-MDT0001: new disk, initializing [12058.201229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12058.227341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12058.237710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12062.729962] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12072.703458] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12072.919919] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12077.362929] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12077.671214] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12083.175666] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12083.178573] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12083.191469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12083.206470] Lustre: Skipped 1 previous similar message [12083.216917] Lustre: Skipped 1 previous similar message [12085.917357] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.7@tcp (stopping) [12088.291403] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12088.297234] Lustre: Skipped 1 previous similar message [12088.897565] Lustre: server umount lustre-MDT0001 complete [12098.529213] LustreError: 435618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.542688] LustreError: 435618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [12103.656685] LustreError: 438383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12103.677922] LustreError: 438383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12106.720802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12106.725572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.738916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12108.770713] LustreError: 435619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.858774] Lustre: server umount lustre-OST0000 complete [12113.895586] LustreError: 435618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12130.271648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12130.740035] Lustre: server umount lustre-MDT0000 complete [12143.690401] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12146.378887] Key type lgssc unregistered [12146.606040] LNet: 440376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12146.611812] LNetError: 440376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12146.633417] LNet: Removed LNI 192.168.204.107@tcp [12147.476961] Key type .llcrypt unregistered [12147.478903] Key type ._llcrypt unregistered [12186.669566] Key type ._llcrypt registered [12186.672630] Key type .llcrypt registered [12186.852356] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [12187.135373] Key type .llcrypt unregistered [12187.137737] Key type ._llcrypt unregistered [12204.192934] Key type ._llcrypt registered [12204.197042] Key type .llcrypt registered [12204.389721] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [12205.806137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12205.836248] alg: No test for adler32 (adler32-zlib) [12206.957788] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [12207.260921] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [12209.008156] Key type lgssc registered [12210.070972] Lustre: Echo OBD driver; http://www.lustre.org/ [12237.525173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12237.575868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.897497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12238.937785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12239.107853] Lustre: lustre-MDT0000: new disk, initializing [12239.197379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12239.239557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12244.294209] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12255.277561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12255.412654] Lustre: 443755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12255.568884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12255.574714] Lustre: Skipped 1 previous similar message [12255.700792] Lustre: lustre-MDT0001: new disk, initializing [12255.844426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12255.895516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12255.913395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12262.019539] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12273.871891] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12282.996510] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12290.608736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12291.048832] Lustre: lustre-OST0000: new disk, initializing [12291.052132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12291.058054] Lustre: 445727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12291.161184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12297.296908] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12301.406131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12301.424082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12301.519953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12306.907241] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12317.155785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12317.158114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12317.169126] Lustre: Skipped 2 previous similar messages [12317.183513] Lustre: Skipped 2 previous similar messages [12320.909627] Lustre: server umount lustre-MDT0000 complete [12324.583582] LustreError: 443748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785232526 with bad export cookie 6993797318486763031 [12324.589746] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12324.592717] LustreError: 443748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12324.881517] Lustre: server umount lustre-MDT0001 complete [12339.425249] Lustre: server umount lustre-OST0000 complete [12351.586582] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:55:51 (1785232551) [12361.065621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12361.678289] LustreError: 447753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12361.696227] LustreError: 447753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12361.757339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12366.826929] LustreError: 447754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12366.986140] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12371.943350] LustreError: 447753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12376.297137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12376.840229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12382.909649] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12392.217334] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12398.975563] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12420.146482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12420.392510] Lustre: Found index 0 for lustre-OST0000, updating log [12420.580969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12428.002788] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [12432.908722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12436.727098] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12443.104101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12443.111111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12443.116508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12443.135157] Lustre: Skipped 1 previous similar message [12444.199300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12444.206879] Lustre: Skipped 2 previous similar messages [12448.299472] Lustre: server umount lustre-OST0000 complete [12464.097818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12464.122204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12464.145067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12470.220949] Lustre: server umount lustre-MDT0000 complete [12470.752664] LustreError: 447748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12470.767731] LustreError: 447748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12474.519870] LustreError: 447733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785232676 with bad export cookie 6993797318486764424 [12474.526544] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12474.531131] LustreError: 447733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12474.955827] Lustre: server umount lustre-MDT0001 complete [12484.090513] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:58:04 (1785232684) [12484.776545] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13115.636980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13116.205117] LustreError: 454185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13116.229347] LustreError: 454185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13116.317238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13121.515059] LustreError: 454186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13121.833890] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13126.636330] LustreError: 454185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.773733] LustreError: 454186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13132.814705] LustreError: 454186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13133.753933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13134.307398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13140.479441] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13150.488030] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13158.965911] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13174.943443] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13181.378586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13181.594048] Lustre: Found index 0 for lustre-OST0000, updating log [13182.075671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13190.333684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [13190.388540] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13199.574782] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13203.946185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13203.960165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13203.989294] Lustre: Skipped 1 previous similar message [13203.998129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13204.006982] Lustre: Skipped 2 previous similar messages [13208.314442] Lustre: server umount lustre-OST0000 complete [13214.440887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13214.449234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13214.482837] Lustre: Skipped 2 previous similar messages [13214.489939] Lustre: Skipped 1 previous similar message [13215.525914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13215.536652] Lustre: Skipped 2 previous similar messages [13217.571369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13217.578565] Lustre: Skipped 2 previous similar messages [13220.049244] Lustre: server umount lustre-MDT0000 complete [13220.707226] LustreError: 454181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.739090] LustreError: 454181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13224.868264] LustreError: 454165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233426 with bad export cookie 6993797318486765509 [13224.871926] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13224.882243] LustreError: 454165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13225.263431] Lustre: server umount lustre-MDT0001 complete [13234.335871] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp [13259.557975] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13274.551505] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13320.942369] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:12:00 (1785233520) [13341.673148] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13345.192663] Key type lgssc unregistered [13345.526661] LNet: 460814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13345.540662] LNetError: 460814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13444.902354] LNet: Removed LNI 192.168.204.107@tcp [13445.718837] Key type .llcrypt unregistered [13445.722669] Key type ._llcrypt unregistered [13463.489342] Key type ._llcrypt registered [13463.494797] Key type .llcrypt registered [13463.613658] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13464.793468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13465.031911] alg: No test for adler32 (adler32-zlib) [13466.046939] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [13466.274763] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13467.944642] Key type lgssc registered [13469.153967] Lustre: Echo OBD driver; http://www.lustre.org/ [13473.541209] LNet: Added LNI 192.168.204.107@tcp100 [8/256/0/180] [13474.601162] LNet: Added LNI 192.168.204.107@tcp101 [8/256/0/180] [13475.541753] LNet: Added LNI 192.168.204.107@tcp102 [8/256/0/180] [13476.569946] LNet: Added LNI 192.168.204.107@tcp103 [8/256/0/180] [13477.529841] LNet: Added LNI 192.168.204.107@tcp104 [8/256/0/180] [13478.398183] LNet: Added LNI 192.168.204.107@tcp105 [8/256/0/180] [13479.409661] LNet: Added LNI 192.168.204.107@tcp106 [8/256/0/180] [13480.408880] LNet: Added LNI 192.168.204.107@tcp107 [8/256/0/180] [13481.508329] LNet: Added LNI 192.168.204.107@tcp108 [8/256/0/180] [13482.502696] LNet: Added LNI 192.168.204.107@tcp109 [8/256/0/180] [13483.590529] LNet: Added LNI 192.168.204.107@tcp110 [8/256/0/180] [13484.536450] LNet: Added LNI 192.168.204.107@tcp111 [8/256/0/180] [13485.781668] LNet: Added LNI 192.168.204.107@tcp112 [8/256/0/180] [13486.690993] LNet: Added LNI 192.168.204.107@tcp113 [8/256/0/180] [13487.855785] LNet: Added LNI 192.168.204.107@tcp114 [8/256/0/180] [13488.889784] LNet: Added LNI 192.168.204.107@tcp115 [8/256/0/180] [13490.049209] LNet: Added LNI 192.168.204.107@tcp116 [8/256/0/180] [13491.072075] LNet: Added LNI 192.168.204.107@tcp117 [8/256/0/180] [13492.144139] LNet: Added LNI 192.168.204.107@tcp118 [8/256/0/180] [13493.149616] LNet: Added LNI 192.168.204.107@tcp119 [8/256/0/180] [13494.349098] LNet: Added LNI 192.168.204.107@tcp120 [8/256/0/180] [13495.593517] LNet: Added LNI 192.168.204.107@tcp121 [8/256/0/180] [13496.690449] LNet: Added LNI 192.168.204.107@tcp122 [8/256/0/180] [13497.707144] LNet: Added LNI 192.168.204.107@tcp123 [8/256/0/180] [13498.663036] LNet: Added LNI 192.168.204.107@tcp124 [8/256/0/180] [13499.715544] LNet: Added LNI 192.168.204.107@tcp125 [8/256/0/180] [13500.791428] LNet: Added LNI 192.168.204.107@tcp126 [8/256/0/180] [13501.829433] LNet: Added LNI 192.168.204.107@tcp127 [8/256/0/180] [13502.715804] LNet: Added LNI 192.168.204.107@tcp128 [8/256/0/180] [13503.448446] LNet: Added LNI 192.168.204.107@tcp129 [8/256/0/180] [13504.279851] LNet: Added LNI 192.168.204.107@tcp130 [8/256/0/180] [13505.123515] LNet: Added LNI 192.168.204.107@tcp131 [8/256/0/180] [13506.060387] LNet: Added LNI 192.168.204.107@tcp132 [8/256/0/180] [13506.942904] LNet: Added LNI 192.168.204.107@tcp133 [8/256/0/180] [13507.861606] LNet: Added LNI 192.168.204.107@tcp134 [8/256/0/180] [13508.915910] LNet: Added LNI 192.168.204.107@tcp135 [8/256/0/180] [13516.183368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13516.228629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13517.337074] Lustre: 463846:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13517.408660] LNetError: 463867:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13517.427467] LNetError: Protocol error connecting to 192.168.204.107@tcp119 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13517.447307] LNetError: Skipped 10 previous similar messages [13517.473139] LNet: 7 local NIs in recovery (showing 7): 192.168.204.107@tcp124, 192.168.204.107@tcp126, 192.168.204.107@tcp125, 192.168.204.107@tcp128, 192.168.204.107@tcp129, 192.168.204.107@tcp127, 192.168.204.107@tcp119 [13517.578489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13517.617499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13517.718263] Lustre: lustre-MDT0000: new disk, initializing [13517.825439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13517.842258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13518.498291] LNetError: 461306:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13518.523371] LNetError: 461306:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13518.530602] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13520.544610] LNetError: 461305:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13520.558176] LNetError: 461305:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13520.578212] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13520.595093] LNetError: Skipped 2 previous similar messages [13523.273924] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13524.644171] LNetError: 463877:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13524.669028] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13532.834382] LNetError: 463873:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13532.863122] LNetError: Protocol error connecting to 192.168.204.107@tcp124 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13535.352613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13535.584661] Lustre: 463863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13535.644201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13535.659726] Lustre: Skipped 1 previous similar message [13535.765570] Lustre: lustre-MDT0001: new disk, initializing [13535.800133] Lustre: 464665:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13535.811545] Lustre: 464665:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13535.921760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13535.972927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13535.981705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13541.387143] Lustre: 464908:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13541.392354] Lustre: 464908:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13541.685812] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13547.169227] LNetError: 463872:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13547.182981] LNetError: Protocol error connecting to 192.168.204.107@tcp120 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13554.578172] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13561.664469] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13564.708890] LNetError: 463873:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13564.733463] LNetError: 463873:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [13564.758230] LNetError: Protocol error connecting to 192.168.204.107@tcp126 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13564.777863] LNetError: Skipped 8 previous similar messages [13570.624964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13570.983505] Lustre: lustre-OST0000: new disk, initializing [13570.987907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13570.994665] Lustre: 465838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13571.135548] Lustre: 465843:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13571.142831] Lustre: 465843:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13571.211026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13577.954536] LNet: 1 local NIs in recovery (showing 1): 192.168.204.107@tcp124 [13579.248882] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13579.789647] Lustre: 466213:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.107@tcp: can't add NID 192.168.204.107@tcp130: rc = -75 [13579.798615] Lustre: 466213:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13579.865689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13579.884374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13580.108360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13590.859939] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13597.485529] LNetError: 463875:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.107 [13597.493370] LNetError: 463875:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [13597.501467] LNetError: Protocol error connecting to 192.168.204.107@tcp125 at host 192.168.204.107:988: is it running a compatible version of Lustre? [13597.516384] LNetError: Skipped 15 previous similar messages [13600.738816] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13600.751266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13600.752977] LustreError: Skipped 1 previous similar message [13600.754432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13600.780806] Lustre: Skipped 1 previous similar message [13605.857401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13605.868888] Lustre: Skipped 1 previous similar message [13606.942856] Lustre: server umount lustre-OST0000 complete [13610.986300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13611.017112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13611.023975] Lustre: Skipped 2 previous similar messages [13616.097250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13616.117701] Lustre: Skipped 1 previous similar message [13621.216702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13621.230836] Lustre: Skipped 1 previous similar message [13625.311324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13625.483716] Lustre: server umount lustre-MDT0000 complete [13629.699115] LustreError: 463855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785233831 with bad export cookie 17954405738179485257 [13629.700827] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13629.714027] LustreError: 463855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13630.079826] Lustre: server umount lustre-MDT0001 complete [13644.825434] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13647.775705] Key type lgssc unregistered [13648.208547] LNet: 467754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13648.219533] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13648.226811] LNet: Removed LNI 192.168.204.107@tcp [13649.249055] LNet: Removed LNI 192.168.204.107@tcp100 [13649.251780] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13649.255703] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13649.261462] LNet: Removed LNI 192.168.204.107@tcp101 [13650.271171] LNet: Removed LNI 192.168.204.107@tcp102 [13650.278473] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13650.289092] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13650.298491] LNet: Removed LNI 192.168.204.107@tcp103 [13651.359416] LNet: Removed LNI 192.168.204.107@tcp104 [13652.383170] LNet: Removed LNI 192.168.204.107@tcp105 [13652.385196] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13652.398567] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13652.409013] LNet: Removed LNI 192.168.204.107@tcp106 [13653.472341] LNet: Removed LNI 192.168.204.107@tcp107 [13654.495167] LNet: Removed LNI 192.168.204.107@tcp108 [13655.520065] LNet: Removed LNI 192.168.204.107@tcp109 [13656.543248] LNet: Removed LNI 192.168.204.107@tcp110 [13656.548511] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13656.554737] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13656.564483] LNet: Removed LNI 192.168.204.107@tcp111 [13657.631950] LNet: Removed LNI 192.168.204.107@tcp112 [13658.655653] LNet: Removed LNI 192.168.204.107@tcp113 [13659.681817] LNet: Removed LNI 192.168.204.107@tcp114 [13660.704528] LNet: Removed LNI 192.168.204.107@tcp115 [13661.729262] LNet: Removed LNI 192.168.204.107@tcp116 [13662.751195] LNet: Removed LNI 192.168.204.107@tcp117 [13663.775323] LNet: Removed LNI 192.168.204.107@tcp118 [13664.801138] LNet: Removed LNI 192.168.204.107@tcp119 [13664.803208] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13664.823228] LNetError: 467754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13664.833734] LNet: Removed LNI 192.168.204.107@tcp120 [13665.887325] LNet: Removed LNI 192.168.204.107@tcp121 [13666.911164] LNet: Removed LNI 192.168.204.107@tcp122 [13667.935545] LNet: Removed LNI 192.168.204.107@tcp123 [13667.937719] LNet: Removed LNI 192.168.204.107@tcp124 [13668.959753] LNet: Removed LNI 192.168.204.107@tcp125 [13669.984647] LNet: Removed LNI 192.168.204.107@tcp126 [13671.008344] LNet: Removed LNI 192.168.204.107@tcp127 [13672.031249] LNet: Removed LNI 192.168.204.107@tcp128 [13673.055386] LNet: Removed LNI 192.168.204.107@tcp129 [13673.058905] LNet: Removed LNI 192.168.204.107@tcp130 [13673.062290] LNet: Removed LNI 192.168.204.107@tcp131 [13673.071112] LNet: Removed LNI 192.168.204.107@tcp132 [13673.074516] LNet: Removed LNI 192.168.204.107@tcp133 [13673.077528] LNet: Removed LNI 192.168.204.107@tcp134 [13673.092969] LNet: Removed LNI 192.168.204.107@tcp135 [13673.987773] Key type .llcrypt unregistered [13673.990753] Key type ._llcrypt unregistered [13699.254667] Key type ._llcrypt registered [13699.256716] Key type .llcrypt registered [13699.343126] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [13717.644565] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13719.122678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13719.185966] alg: No test for adler32 (adler32-zlib) [13720.400760] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [13720.805160] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13722.527255] Key type lgssc registered [13724.830751] Lustre: Echo OBD driver; http://www.lustre.org/ [13761.476306] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:19:21 (1785233961) [13778.420392] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [13781.403391] Key type lgssc unregistered [13781.796500] LNet: 471848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13781.807949] LNetError: 471848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13781.825790] LNet: Removed LNI 192.168.204.107@tcp [13782.834977] Key type .llcrypt unregistered [13782.837800] Key type ._llcrypt unregistered [13802.128844] Key type ._llcrypt registered [13802.130777] Key type .llcrypt registered [13802.246656] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [13803.142298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13803.256287] alg: No test for adler32 (adler32-zlib) [13804.367598] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [13804.629455] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [13806.295192] Key type lgssc registered [13807.188085] Lustre: Echo OBD driver; http://www.lustre.org/ [13816.829789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13816.852566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13818.104848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13818.136452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13818.214922] Lustre: lustre-MDT0000: new disk, initializing [13818.316359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13818.348233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13823.427673] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13833.739569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13833.890829] Lustre: 473061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13833.950439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13833.957680] Lustre: Skipped 1 previous similar message [13833.991875] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13834.059839] Lustre: lustre-MDT0001: new disk, initializing [13834.141701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13834.177705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13834.186287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13838.916043] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13847.961736] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13853.762740] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13861.482622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13861.790558] Lustre: lustre-OST0000: new disk, initializing [13861.795799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13861.803903] Lustre: 475029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13861.880436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13869.673743] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [13871.687364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13871.707072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13871.841439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13880.214425] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13884.859615] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing lnet_if_list [13890.432065] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13890.464624] LNet: Added LNI 192.168.204.107@tcp10 [8/256/0/180] [13897.719410] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13897.779469] LNet: Added LNI 192.168.204.107@tcp11 [8/256/0/180] [13903.197578] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13903.268856] LNet: Added LNI 192.168.204.107@tcp12 [8/256/0/180] [13909.454273] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13909.489932] LNet: Added LNI 192.168.204.107@tcp13 [8/256/0/180] [13915.510171] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13915.539173] LNet: Added LNI 192.168.204.107@tcp14 [8/256/0/180] [13920.840763] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13920.887178] LNet: Added LNI 192.168.204.107@tcp15 [8/256/0/180] [13927.574613] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13927.616789] LNet: Added LNI 192.168.204.107@tcp16 [8/256/0/180] [13933.624898] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13933.661336] LNet: Added LNI 192.168.204.107@tcp17 [8/256/0/180] [13939.042284] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13939.079045] LNet: Added LNI 192.168.204.107@tcp18 [8/256/0/180] [13944.408851] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13944.466104] LNet: Added LNI 192.168.204.107@tcp19 [8/256/0/180] [13949.792192] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13949.828427] LNet: Added LNI 192.168.204.107@tcp20 [8/256/0/180] [13955.150117] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13955.185791] LNet: Added LNI 192.168.204.107@tcp21 [8/256/0/180] [13960.787940] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13960.825928] LNet: Added LNI 192.168.204.107@tcp22 [8/256/0/180] [13966.855113] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13966.907710] LNet: Added LNI 192.168.204.107@tcp23 [8/256/0/180] [13973.608577] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13973.641723] LNet: Added LNI 192.168.204.107@tcp24 [8/256/0/180] [13979.928911] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13979.965264] LNet: Added LNI 192.168.204.107@tcp25 [8/256/0/180] [13986.076717] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13986.118140] LNet: Added LNI 192.168.204.107@tcp26 [8/256/0/180] [13992.281162] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13992.320878] LNet: Added LNI 192.168.204.107@tcp27 [8/256/0/180] [13999.406586] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13999.451658] LNet: Added LNI 192.168.204.107@tcp28 [8/256/0/180] [14006.072755] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14006.106613] LNet: Added LNI 192.168.204.107@tcp29 [8/256/0/180] [14012.618709] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14012.659391] LNet: Added LNI 192.168.204.107@tcp30 [8/256/0/180] [14018.840509] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14018.888457] LNet: Added LNI 192.168.204.107@tcp31 [8/256/0/180] [14025.711298] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14025.740365] LNet: Added LNI 192.168.204.107@tcp32 [8/256/0/180] [14032.697912] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14032.726525] LNet: Added LNI 192.168.204.107@tcp33 [8/256/0/180] [14039.978392] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14040.029718] LNet: Added LNI 192.168.204.107@tcp34 [8/256/0/180] [14046.569576] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14046.603034] LNet: Added LNI 192.168.204.107@tcp35 [8/256/0/180] [14054.203220] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14054.233883] LNet: Added LNI 192.168.204.107@tcp36 [8/256/0/180] [14062.744537] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14062.785751] LNet: Added LNI 192.168.204.107@tcp37 [8/256/0/180] [14070.536501] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14070.569472] LNet: Added LNI 192.168.204.107@tcp38 [8/256/0/180] [14078.281290] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14078.320506] LNet: Added LNI 192.168.204.107@tcp39 [8/256/0/180] [14085.951980] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14085.980125] LNet: Added LNI 192.168.204.107@tcp40 [8/256/0/180] [14093.600987] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14093.620380] LNet: Added LNI 192.168.204.107@tcp41 [8/256/0/180] [14101.500424] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14101.565542] LNet: Added LNI 192.168.204.107@tcp42 [8/256/0/180] [14110.432948] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14110.461512] LNet: Added LNI 192.168.204.107@tcp43 [8/256/0/180] [14119.124782] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14119.154922] LNet: Added LNI 192.168.204.107@tcp44 [8/256/0/180] [14127.522840] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14127.596505] LNet: Added LNI 192.168.204.107@tcp45 [8/256/0/180] [14135.903787] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14135.928125] LNet: Added LNI 192.168.204.107@tcp46 [8/256/0/180] [14144.321988] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14144.356104] LNet: Added LNI 192.168.204.107@tcp47 [8/256/0/180] [14153.392633] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14153.448049] LNet: Added LNI 192.168.204.107@tcp48 [8/256/0/180] [14161.944452] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14161.990514] LNet: Added LNI 192.168.204.107@tcp49 [8/256/0/180] [14172.318504] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14172.381361] LNet: Added LNI 192.168.204.107@tcp50 [8/256/0/180] [14192.619754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14192.625782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14192.648552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14194.658341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14194.661303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14194.689214] Lustre: Skipped 2 previous similar messages [14198.955720] Lustre: server umount lustre-MDT0000 complete [14204.126481] LustreError: 473052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234406 with bad export cookie 17089465780715639515 [14204.128828] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14204.156827] LustreError: 473052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14204.729109] Lustre: server umount lustre-MDT0001 complete [14220.255142] Lustre: 472334:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785234406/real 1785234406] req@00000000e9431ed4 x1871953533256064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785234422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14220.288597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14220.311130] Lustre: Skipped 1 previous similar message [14222.018518] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14225.375107] Lustre: 472332:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785234411/real 1785234411] req@0000000029d4de2f x1871953533256192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785234427 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14229.994091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14230.560042] Lustre: 472332:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785234416/real 1785234416] req@0000000039c40e40 x1871953533256448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785234432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14235.616020] Lustre: 472334:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785234421/real 1785234421] req@00000000ff5e1aa2 x1871953533256576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785234437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14250.271241] LustreError: 475021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14250.294229] LustreError: 475021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14250.335808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14254.514691] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14255.587045] LustreError: 475022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14260.726991] LustreError: 475021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14261.885627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14262.274407] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14265.380615] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14266.676862] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14273.670524] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14274.668714] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14279.108987] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14300.649025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14300.659836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14300.683063] Lustre: Skipped 1 previous similar message [14300.697618] Lustre: Skipped 1 previous similar message [14304.377369] Lustre: server umount lustre-OST0000 complete [14308.320400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14308.326673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14308.340751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14314.203872] Lustre: server umount lustre-MDT0000 complete [14316.000101] LustreError: 499274:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14316.014547] LustreError: 499274:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14317.405639] LustreError: 473051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234519 with bad export cookie 17089465780715642224 [14317.414581] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14317.416113] LustreError: 473051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14317.708840] Lustre: server umount lustre-MDT0001 complete [14326.320245] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14328.689883] Key type lgssc unregistered [14329.114930] LNet: 502068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14329.127103] LNetError: 502068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14329.136764] LNet: Removed LNI 192.168.204.107@tcp [14329.141932] LNet: Removed LNI 192.168.204.107@tcp10 [14329.147159] LNet: Removed LNI 192.168.204.107@tcp11 [14329.151251] LNet: Removed LNI 192.168.204.107@tcp12 [14329.154663] LNet: Removed LNI 192.168.204.107@tcp13 [14329.160529] LNet: Removed LNI 192.168.204.107@tcp14 [14329.163820] LNet: Removed LNI 192.168.204.107@tcp15 [14329.167424] LNet: Removed LNI 192.168.204.107@tcp16 [14329.178187] LNet: Removed LNI 192.168.204.107@tcp17 [14329.189812] LNet: Removed LNI 192.168.204.107@tcp18 [14329.193191] LNet: Removed LNI 192.168.204.107@tcp19 [14329.196770] LNet: Removed LNI 192.168.204.107@tcp20 [14329.200578] LNet: Removed LNI 192.168.204.107@tcp21 [14329.203867] LNet: Removed LNI 192.168.204.107@tcp22 [14329.207727] LNet: Removed LNI 192.168.204.107@tcp23 [14329.215501] LNet: Removed LNI 192.168.204.107@tcp24 [14329.220127] LNet: Removed LNI 192.168.204.107@tcp25 [14329.224480] LNet: Removed LNI 192.168.204.107@tcp26 [14329.229158] LNet: Removed LNI 192.168.204.107@tcp27 [14329.233841] LNet: Removed LNI 192.168.204.107@tcp28 [14329.242576] LNet: Removed LNI 192.168.204.107@tcp29 [14329.247105] LNet: Removed LNI 192.168.204.107@tcp30 [14329.250544] LNet: Removed LNI 192.168.204.107@tcp31 [14329.255810] LNet: Removed LNI 192.168.204.107@tcp32 [14329.258916] LNet: Removed LNI 192.168.204.107@tcp33 [14329.263355] LNet: Removed LNI 192.168.204.107@tcp34 [14329.270059] LNet: Removed LNI 192.168.204.107@tcp35 [14329.275572] LNet: Removed LNI 192.168.204.107@tcp36 [14329.279156] LNet: Removed LNI 192.168.204.107@tcp37 [14329.281310] LNet: Removed LNI 192.168.204.107@tcp38 [14329.283929] LNet: Removed LNI 192.168.204.107@tcp39 [14329.287319] LNet: Removed LNI 192.168.204.107@tcp40 [14329.289860] LNet: Removed LNI 192.168.204.107@tcp41 [14329.293107] LNet: Removed LNI 192.168.204.107@tcp42 [14329.299122] LNet: Removed LNI 192.168.204.107@tcp43 [14329.302881] LNet: Removed LNI 192.168.204.107@tcp44 [14329.305610] LNet: Removed LNI 192.168.204.107@tcp45 [14329.309866] LNet: Removed LNI 192.168.204.107@tcp46 [14329.313861] LNet: Removed LNI 192.168.204.107@tcp47 [14329.317457] LNet: Removed LNI 192.168.204.107@tcp48 [14329.326152] LNet: Removed LNI 192.168.204.107@tcp49 [14329.344219] LNet: Removed LNI 192.168.204.107@tcp50 [14330.156561] Key type .llcrypt unregistered [14330.164423] Key type ._llcrypt unregistered [14350.879188] Key type ._llcrypt registered [14350.881527] Key type .llcrypt registered [14350.963470] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14367.041143] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14368.073834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14368.215666] alg: No test for adler32 (adler32-zlib) [14369.656740] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [14370.089874] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14371.831207] Key type lgssc registered [14373.245985] Lustre: Echo OBD driver; http://www.lustre.org/ [14412.313428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14412.352385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14413.575621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14413.610788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14413.660527] Lustre: lustre-MDT0000: new disk, initializing [14413.721542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14413.741679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14418.221199] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14428.747495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14428.866780] Lustre: 505267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14428.935961] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14428.944711] Lustre: Skipped 1 previous similar message [14429.037630] Lustre: lustre-MDT0001: new disk, initializing [14429.157874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14429.196047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14429.200609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14432.969586] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14441.672812] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14447.862475] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14455.934263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14456.102210] Lustre: *** cfs_fail_loc=90f, val=0*** [14456.172721] Lustre: lustre-OST0000: new disk, initializing [14456.177638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14456.186850] Lustre: 507283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14456.283901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14462.466504] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14465.602399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14465.621914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14465.710406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14472.162516] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14481.382207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14481.383752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14481.390310] LustreError: Skipped 1 previous similar message [14481.394659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14481.408782] Lustre: Skipped 1 previous similar message [14485.439172] Lustre: server umount lustre-OST0000 complete [14490.595473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14490.604148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14490.618291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14490.625310] Lustre: Skipped 1 previous similar message [14491.618315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14491.639172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14491.654889] Lustre: Skipped 1 previous similar message [14495.883416] Lustre: server umount lustre-MDT0000 complete [14500.024098] LustreError: 505259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234701 with bad export cookie 11425884146852272694 [14500.029084] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14500.035613] LustreError: 505259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14500.466991] Lustre: server umount lustre-MDT0001 complete [14512.795317] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14515.587320] Key type lgssc unregistered [14515.848781] LNet: 509189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14515.859370] LNetError: 509189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14515.891489] LNet: Removed LNI 192.168.204.107@tcp [14516.903594] Key type .llcrypt unregistered [14516.908102] Key type ._llcrypt unregistered [14539.373959] Key type ._llcrypt registered [14539.378098] Key type .llcrypt registered [14539.515902] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14540.705834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14540.815538] alg: No test for adler32 (adler32-zlib) [14542.058357] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [14542.319027] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14544.007166] Key type lgssc registered [14545.442216] Lustre: Echo OBD driver; http://www.lustre.org/ [14554.113586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14554.128369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14555.660320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14560.978280] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14565.860863] LustreError: 510457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14565.872879] LustreError: 510457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14570.734992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14570.979352] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [14571.288562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14577.060869] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14585.673381] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14591.592958] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14598.364881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14598.724543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14605.690823] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14613.943696] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14627.523767] loop0: detected capacity change from 0 to 204800000 [14627.720977] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14627.731632] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14627.740398] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14635.438070] loop0: detected capacity change from 0 to 204800000 [14635.721095] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14635.736525] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14635.761940] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14641.440277] loop0: detected capacity change from 0 to 204800000 [14647.892490] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14648.038446] Lustre: fs15246-OST0000: new disk, initializing [14648.042503] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14648.047895] Lustre: 514054:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14648.117906] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14654.234398] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14663.701904] loop1: detected capacity change from 0 to 204800000 [14669.853030] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14669.969851] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14670.026245] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14670.131425] Lustre: fs15246-MDT0000: new disk, initializing [14670.349677] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14670.386175] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14670.535083] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14671.585869] Lustre: 515175:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000057d99ba9 x1871954306973184/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785234889 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14675.285853] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14680.676192] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14680.691443] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14680.737349] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14698.474825] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14698.490299] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14703.461057] Lustre: server umount fs15246-MDT0000 complete [14703.595844] LustreError: 510453:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14708.708549] LustreError: 513377:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14713.833845] LustreError: 510452:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14714.045399] Lustre: server umount fs15246-OST0000 complete [14729.195673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14729.201161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14729.202123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14729.220995] Lustre: Skipped 1 previous similar message [14731.979480] Lustre: server umount lustre-OST0000 complete [14735.330047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14735.339276] LustreError: Skipped 1 previous similar message [14735.347772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14735.360709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14735.365894] Lustre: Skipped 1 previous similar message [14739.433271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14739.446638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14741.619345] Lustre: server umount lustre-MDT0000 complete [14744.544842] LustreError: 511185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14744.586613] LustreError: 511185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14745.964600] LustreError: 510438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785234947 with bad export cookie 2367997576987995544 [14745.971356] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14745.978920] LustreError: 510438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14746.371545] Lustre: server umount lustre-MDT0001 complete [14759.865253] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [14762.563685] Key type lgssc unregistered [14762.832881] LNet: 518276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14762.841395] LNetError: 518276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14762.859616] LNet: Removed LNI 192.168.204.107@tcp [14763.715856] Key type .llcrypt unregistered [14763.719394] Key type ._llcrypt unregistered [14782.438599] Key type ._llcrypt registered [14782.441955] Key type .llcrypt registered [14782.568891] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14783.766309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14783.847683] alg: No test for adler32 (adler32-zlib) [14784.950506] Lustre: Lustre: Build Version: 2.17.54_204_g5db9db4 [14785.158492] LNet: Added LNI 192.168.204.107@tcp [8/256/0/180] [14786.807178] Key type lgssc registered [14788.129692] Lustre: Echo OBD driver; http://www.lustre.org/ [14830.572460] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [14843.058776] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [14880.874685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14880.899446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14882.129309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14882.173569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14882.270272] Lustre: lustre-MDT0000: new disk, initializing [14882.385254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14882.404430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14886.762849] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14898.078362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14898.211469] Lustre: 522746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14898.289302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14898.297531] Lustre: Skipped 1 previous similar message [14898.383585] Lustre: lustre-MDT0001: new disk, initializing [14898.487960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14898.534029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14898.553167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14903.149699] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14912.953726] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14918.632927] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14925.781838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14926.079842] Lustre: lustre-OST0000: new disk, initializing [14926.085481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14926.094509] Lustre: 524713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14926.195620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14932.962841] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_default_debug -1 all [14934.044581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14934.051486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14934.153468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14941.971584] Lustre: DEBUG MARKER: oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14947.610757] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14947.843151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14952.843838] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14953.163700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14959.585789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14959.596930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14959.613541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14960.109296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14960.156018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14960.173961] Lustre: Skipped 1 previous similar message [14961.865967] Lustre: server umount lustre-OST0000 complete [14970.339975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14970.342689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14970.350419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14970.374515] Lustre: Skipped 1 previous similar message [14972.423505] Lustre: server umount lustre-MDT0000 complete [14976.878027] LustreError: 522738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785235178 with bad export cookie 8670700975533366887 [14976.883555] LustreError: MGC192.168.204.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14976.898933] LustreError: 522738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14977.224672] Lustre: server umount lustre-MDT0001 complete [15005.620995] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing set_hostid [15017.742585] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing load_modules_local [15052.805637] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14737 sec ========= 06:40:52 (1785235252) [15055.030518] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:40:54 (1785235254) === [15058.393725] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:40:58 (1785235258) === [15108.405274] Lustre: DEBUG MARKER: oleg407-server.virtnet: executing unload_modules_local [15111.991888] Key type lgssc unregistered [15112.411959] LNet: 531360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15112.425108] LNetError: 531360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15112.443982] LNet: Removed LNI 192.168.204.107@tcp [15113.362304] Key type .llcrypt unregistered [15113.368493] Key type ._llcrypt unregistered