[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485217672 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006940] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012065] *** VALIDATE selinux *** [ 0.021223] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026717] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027174] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030124] *** VALIDATE tmpfs *** [ 0.031505] *** VALIDATE proc *** [ 0.032285] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042420] debug: unmapping init [mem 0xffffffffbb459000-0xffffffffbb460fff] [ 0.044797] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046026] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047038] ... version: 2 [ 0.048022] ... bit width: 48 [ 0.049020] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052023] ... fixed-purpose events: 3 [ 0.053035] ... event mask: 000000070000000f [ 0.054453] rcu: Hierarchical SRCU implementation. [ 0.057427] smp: Bringing up secondary CPUs ... [ 0.058833] x86: Booting SMP configuration: [ 0.059056] .... node #0, CPUs: #1 #2 #3 [ 0.063031] smp: Brought up 1 node, 4 CPUs [ 0.065041] smpboot: Max logical packages: 1 [ 0.066023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.184981] node 0 deferred pages initialised in 117ms [ 0.189220] devtmpfs: initialized [ 0.191351] x86/mm: Memory block size: 128MB [ 0.195528] gcov: version magic: 0x41383552 [ 0.198345] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.203120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.206326] pinctrl core: initialized pinctrl subsystem [ 0.209213] [ 0.209926] ************************************************************* [ 0.212023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.215021] ** ** [ 0.217015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.220018] ** ** [ 0.222022] ** This means that this kernel is built to expose internal ** [ 0.225024] ** IOMMU data structures, which may compromise security on ** [ 0.228019] ** your system. ** [ 0.230018] ** ** [ 0.233021] ** If you see this message and you are not debugging the ** [ 0.236022] ** kernel, report this immediately to your vendor! ** [ 0.239018] ** ** [ 0.241016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244019] ************************************************************* [ 0.247655] NET: Registered protocol family 16 [ 0.249469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.253096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.256111] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.260121] cpuidle: using governor menu [ 0.261816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.262629] PCI: Using configuration type 1 for base access [ 0.265146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.277061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.278030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280079] cryptd: max_cpu_qlen set to 1000 [ 0.284139] ACPI: Added _OSI(Module Device) [ 0.285020] ACPI: Added _OSI(Processor Device) [ 0.287016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.293538] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.301458] ACPI: Interpreter enabled [ 0.303073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.305018] ACPI: Using IOAPIC for interrupt routing [ 0.307155] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311556] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.321000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.324057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.326022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.330117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334627] acpiphp: Slot [2] registered [ 0.336187] acpiphp: Slot [5] registered [ 0.337194] acpiphp: Slot [6] registered [ 0.339174] acpiphp: Slot [7] registered [ 0.341278] acpiphp: Slot [8] registered [ 0.342215] acpiphp: Slot [9] registered [ 0.344187] acpiphp: Slot [10] registered [ 0.346181] acpiphp: Slot [3] registered [ 0.347117] acpiphp: Slot [4] registered [ 0.348155] acpiphp: Slot [11] registered [ 0.350110] acpiphp: Slot [12] registered [ 0.351139] acpiphp: Slot [13] registered [ 0.353113] acpiphp: Slot [14] registered [ 0.354108] acpiphp: Slot [15] registered [ 0.356103] acpiphp: Slot [16] registered [ 0.357100] acpiphp: Slot [17] registered [ 0.358142] acpiphp: Slot [18] registered [ 0.360138] acpiphp: Slot [19] registered [ 0.361265] acpiphp: Slot [20] registered [ 0.362109] acpiphp: Slot [21] registered [ 0.364103] acpiphp: Slot [22] registered [ 0.365132] acpiphp: Slot [23] registered [ 0.366102] acpiphp: Slot [24] registered [ 0.368106] acpiphp: Slot [25] registered [ 0.369147] acpiphp: Slot [26] registered [ 0.370119] acpiphp: Slot [27] registered [ 0.372146] acpiphp: Slot [28] registered [ 0.374134] acpiphp: Slot [29] registered [ 0.375133] acpiphp: Slot [30] registered [ 0.377157] acpiphp: Slot [31] registered [ 0.379094] PCI host bridge to bus 0000:00 [ 0.381031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.384034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.386028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.389034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.392094] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.395038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.397202] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.401190] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.405159] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.416798] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.421880] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.425028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.428023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.431025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.434959] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439106] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.441070] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.445058] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.451019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.463025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.468012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.474833] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.480022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.487018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.507020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.515871] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.525021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.533020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.564607] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.576021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.585019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.598019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.610567] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.618014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.625017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.644031] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.657820] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.673017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.694030] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.707762] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.714028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.724019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.740018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.751256] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.754480] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.757513] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.760528] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.763372] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.769244] iommu: Default domain type: Passthrough [ 0.774566] SCSI subsystem initialized [ 0.776205] ACPI: bus type USB registered [ 0.777156] usbcore: registered new interface driver usbfs [ 0.779122] usbcore: registered new interface driver hub [ 0.781147] usbcore: registered new device driver usb [ 0.784207] pps_core: LinuxPPS API ver. 1 registered [ 0.786015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.789081] PTP clock support registered [ 0.792101] EDAC MC: Ver: 3.0.0 [ 0.794134] PCI: Using ACPI for IRQ routing [ 0.796964] NetLabel: Initializing [ 0.798016] NetLabel: domain hash size = 128 [ 0.800022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.802144] NetLabel: unlabeled traffic allowed by default [ 0.805121] vgaarb: loaded [ 0.807336] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.809022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.817015] clocksource: Switched to clocksource kvm-clock [ 0.928416] VFS: Disk quotas dquot_6.6.0 [ 0.931082] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.934559] *** VALIDATE ramfs *** [ 0.936269] *** VALIDATE hugetlbfs *** [ 0.938246] pnp: PnP ACPI init [ 0.941122] pnp: PnP ACPI: found 6 devices [ 0.966417] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.970303] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.972417] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.974794] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.977437] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.980237] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.983361] NET: Registered protocol family 2 [ 0.986336] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.991554] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.995632] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.001539] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.004771] TCP: Hash tables configured (established 65536 bind 65536) [ 1.007805] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.011319] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.013791] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016785] NET: Registered protocol family 1 [ 1.018827] RPC: Registered named UNIX socket transport module. [ 1.020969] RPC: Registered udp transport module. [ 1.022752] RPC: Registered tcp transport module. [ 1.024690] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.027340] NET: Registered protocol family 44 [ 1.029167] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.031651] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034118] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.036715] PCI: CLS 0 bytes, default 64 [ 1.038632] Unpacking initramfs... [ 2.525155] debug: unmapping init [mem 0xffff8b307cc54000-0xffff8b307ffbffff] [ 2.532136] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.534778] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.539584] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.046108] Initialise system trusted keyrings [ 3.048060] Key type blacklist registered [ 3.050182] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.059466] zbud: loaded [ 3.062540] *** VALIDATE nfs *** [ 3.063948] *** VALIDATE nfs4 *** [ 3.065742] pstore: using deflate compression [ 3.069307] Platform Keyring initialized [ 3.181042] NET: Registered protocol family 38 [ 3.182931] Key type asymmetric registered [ 3.185079] Asymmetric key parser 'x509' registered [ 3.187224] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.190493] io scheduler mq-deadline registered [ 3.192462] io scheduler kyber registered [ 3.194310] io scheduler bfq registered [ 3.196369] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.199809] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.204187] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.207264] ACPI: Power Button [PWRF] [ 3.212814] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.221801] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.243302] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.252159] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.269763] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.297068] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.326017] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.331633] Non-volatile memory driver v1.3 [ 3.333530] Linux agpgart interface v0.103 [ 3.365681] virtio_blk virtio1: [vda] 147224 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.369550] vda: detected capacity change from 0 to 75378688 [ 3.386575] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.390090] vdb: detected capacity change from 0 to 1073741824 [ 3.405570] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.409262] vdc: detected capacity change from 0 to 2621440000 [ 3.429404] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.433322] vdd: detected capacity change from 0 to 2621440000 [ 3.450580] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.453619] vde: detected capacity change from 0 to 4294967296 [ 3.469310] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.472321] vdf: detected capacity change from 0 to 4294967296 [ 3.479524] libphy: Fixed MDIO Bus: probed [ 3.485963] usbcore: registered new interface driver usbserial_generic [ 3.488406] usbserial: USB Serial support registered for generic [ 3.491452] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.496379] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.498650] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.501585] mousedev: PS/2 mouse device common for all mice [ 3.504632] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.507423] rtc_cmos 00:05: RTC can wake from S4 [ 3.514107] rtc_cmos 00:05: registered as rtc0 [ 3.518157] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.521910] intel_pstate: CPU model not supported [ 3.526703] hid: raw HID events driver (C) Jiri Kosina [ 3.527075] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.531393] usbcore: registered new interface driver usbhid [ 3.531401] usbhid: USB HID core driver [ 3.531559] drop_monitor: Initializing network drop monitor service [ 3.531689] Initializing XFRM netlink socket [ 3.532761] NET: Registered protocol family 10 [ 3.546607] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.548079] Segment Routing with IPv6 [ 3.553575] NET: Registered protocol family 17 [ 3.555396] mpls_gso: MPLS GSO support [ 3.561765] RAS: Correctable Errors collector initialized. [ 3.564449] AVX version of gcm_enc/dec engaged. [ 3.566149] AES CTR mode by8 optimization enabled [ 3.663607] sched_clock: Marking stable (3663518089, 0)->(4604754827, -941236738) [ 3.667297] registered taskstats version 1 [ 3.669102] Loading compiled-in X.509 certificates [ 3.670859] zswap: loaded using pool lzo/zbud [ 3.697705] Key type big_key registered [ 3.709857] Key type encrypted registered [ 3.711587] ima: No TPM chip found, activating TPM-bypass! [ 3.713504] ima: Allocated hash algorithm: sha1 [ 3.715151] ima: No architecture policies found [ 3.716809] evm: Initialising EVM extended attributes: [ 3.719982] evm: security.selinux [ 3.722193] evm: security.ima [ 3.723309] evm: security.capability [ 3.724471] evm: HMAC attrs: 0x1 [ 3.726866] rtc_cmos 00:05: setting system clock to 2026-09-11 11:59:17 UTC (1789127957) [ 3.732849] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.735431] debug: unmapping init [mem 0xffffffffbb182000-0xffffffffbb458fff] [ 3.744123] Write protecting the kernel read-only data: 28672k [ 3.747747] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 3.750355] debug: unmapping init [mem 0xffffffffba114000-0xffffffffba1fffff] [ 3.783439] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.790048] systemd[1]: Detected virtualization kvm. [ 3.791598] systemd[1]: Detected architecture x86-64. [ 3.793275] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.820215] systemd[1]: No hostname configured. [ 3.821984] systemd[1]: Set hostname to . [ 3.824033] random: systemd: uninitialized urandom read (16 bytes read) [ 3.826314] systemd[1]: Initializing machine ID from random generator. [ 3.878366] random: ln: uninitialized urandom read (6 bytes read) [ 3.975349] random: systemd: uninitialized urandom read (16 bytes read) [ 3.977386] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.984955] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.990179] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.611218] device-mapper: uevent: version 1.0.3 [ 4.613579] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.285482] random: fast init done [ 5.295052] virtio_net virtio0 ens2: renamed from eth0 [ 5.343392] scsi host0: ata_piix [ 5.370424] scsi host1: ata_piix [ 5.371725] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.373806] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.654374] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.229449] random: crng init done [ 10.230820] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.653304] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.786852] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.067659] SELinux: Disabled at runtime. [ 12.132173] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.139581] systemd[1]: Detected virtualization kvm. [ 12.141568] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.675728] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.679811] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.687408] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.692675] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.695461] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.702262] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.706142] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK [[ 12.858398] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.125496] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.485789] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.496389] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.629275] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.639501] EDAC sbridge: Ver: 1.1.2 [ 15.247979] Key type dns_resolver registered [ 15.547190] NFS: Registering the id_resolver key type [ 15.549275] Key type id_resolver registered [ 15.550717] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg602-server login: [ 41.845501] libcfs: loading out-of-tree module taints kernel. [ 41.866788] Key type ._llcrypt registered [ 41.868325] Key type .llcrypt registered [ 41.925657] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 51.309501] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 51.895528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.903572] alg: No test for adler32 (adler32-zlib) [ 52.944282] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 53.306178] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 54.967218] Key type lgssc registered [ 56.774991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.872586] hrtimer: interrupt took 2574297 ns [ 76.262718] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 115.373271] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 128.623170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 128.646307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 129.939363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 129.978844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 130.067878] Lustre: lustre-MDT0000: new disk, initializing [ 130.223965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 130.246395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 135.946374] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.569766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 152.708141] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 152.764644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 152.770661] Lustre: Skipped 1 previous similar message [ 152.907505] Lustre: lustre-MDT0001: new disk, initializing [ 153.019711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.063825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.087811] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 159.002377] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.648129] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.497284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 176.759544] Lustre: lustre-OST0000: new disk, initializing [ 176.763810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 176.770562] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 176.826539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.885658] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.362766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 184.371652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 184.471255] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.436578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.580609] Lustre: lustre-OST0001: new disk, initializing [ 197.585795] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.594356] Lustre: 9514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.686810] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.878680] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.889191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 204.905099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 204.972628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 216.443625] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.320771] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 235.221688] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing check_logdir /tmp/testlogs/ [ 240.331387] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing yml_node [ 244.076952] Lustre: DEBUG MARKER: Client: 2.17.57.135 [ 246.083353] Lustre: DEBUG MARKER: MDS: 2.17.57.135 [ 248.287891] Lustre: DEBUG MARKER: OSS: 2.17.57.135 [ 249.870281] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 08:03:21 EDT 2026 [ 268.267733] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 270.193934] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 290.272318] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.275458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 290.292029] Lustre: Skipped 1 previous similar message [ 290.310658] Lustre: Skipped 3 previous similar messages [ 292.893989] Lustre: server umount lustre-MDT0000 complete [ 296.965180] LustreError: 10330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128250 with bad export cookie 9297624593469503190 [ 296.973346] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 296.994872] LustreError: 10330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 300.512108] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.523181] Lustre: Skipped 3 previous similar messages [ 300.523270] LustreError: 10506:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 300.524350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 300.566211] LustreError: 10506:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 301.535869] LustreError: 10139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.538497] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 301.560513] LustreError: 10139:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 301.576706] Lustre: Skipped 2 previous similar messages [ 303.406520] Lustre: server umount lustre-MDT0001 complete [ 306.650495] Lustre: server umount lustre-OST0000 complete [ 309.757897] Lustre: server umount lustre-OST0001 complete [ 315.871521] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 325.438217] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 356.704125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.994500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 357.034439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 357.128649] Lustre: lustre-MDT0000: new disk, initializing [ 357.199081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 357.211429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.832701] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 371.738318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.836151] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 371.885848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 371.890382] Lustre: Skipped 1 previous similar message [ 371.995661] Lustre: lustre-MDT0001: new disk, initializing [ 372.125467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 372.182341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 372.191327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 376.554596] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 385.370493] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.773864] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 398.368349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 398.568122] Lustre: lustre-OST0000: new disk, initializing [ 398.571515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 398.576328] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 398.645414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.862095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 399.875391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 399.987949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.429781] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 414.682992] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 419.423509] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 419.668495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 423.890671] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 424.094428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 430.566228] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 430.566745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.579161] LustreError: Skipped 1 previous similar message [ 430.595733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.682343] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.687476] Lustre: Skipped 2 previous similar messages [ 440.288092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 440.443193] Lustre: server umount lustre-OST0000 complete [ 443.873280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 443.883595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.899700] Lustre: Skipped 1 previous similar message [ 443.905747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.207178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 458.406369] Lustre: server umount lustre-MDT0000 complete [ 460.257078] LustreError: 15079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.285954] LustreError: 15079:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 461.563145] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128415 with bad export cookie 9297624593469504723 [ 461.577609] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 461.889605] Lustre: server umount lustre-MDT0001 complete [ 465.685650] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:06:57 (1789128417) [ 467.875372] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 469.986571] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:07:01 (1789128421) [ 475.552877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.077400] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 476.154914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.543462] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 481.256523] LustreError: 19451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 489.856860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.685894] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 503.137218] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 508.930942] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 516.324671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.365890] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 532.532965] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 546.272504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 546.279979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 546.292088] LustreError: Skipped 1 previous similar message [ 546.303601] Lustre: Skipped 1 previous similar message [ 546.309529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 546.323215] Lustre: Skipped 7 previous similar messages [ 549.492762] Lustre: server umount lustre-OST0000 complete [ 556.516200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.528689] Lustre: Skipped 1 previous similar message [ 559.329696] Lustre: server umount lustre-MDT0000 complete [ 561.634867] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.662114] LustreError: 19450:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 563.684684] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128517 with bad export cookie 9297624593469505605 [ 563.686763] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.694019] LustreError: 19432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 564.475735] Lustre: server umount lustre-MDT0001 complete [ 580.123990] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 584.337218] Key type lgssc unregistered [ 584.864907] LNet: 22988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.881026] LNetError: 22988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.904901] LNet: Removed LNI 192.168.206.102@tcp [ 586.315138] Key type .llcrypt unregistered [ 586.317961] Key type ._llcrypt unregistered [ 610.156474] Key type ._llcrypt registered [ 610.165660] Key type .llcrypt registered [ 610.333324] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 611.877957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 611.932090] alg: No test for adler32 (adler32-zlib) [ 612.981608] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 613.259945] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 614.927167] Key type lgssc registered [ 616.568481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 626.464754] loop: module loaded [ 626.601240] loop0: detected capacity change from 0 to 204800000 [ 626.930573] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.951788] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.398854] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.935527] loop0: detected capacity change from 0 to 204800000 [ 633.125246] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.136258] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.145311] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.357494] loop0: detected capacity change from 0 to 204800000 [ 645.300935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 645.337725] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.693762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 646.760179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 646.894957] Lustre: lustre-MDT0000: new disk, initializing [ 646.999843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.023566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 651.800330] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 661.073879] loop1: detected capacity change from 0 to 204800000 [ 666.283811] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 666.599631] Lustre: lustre-OST07c6: new disk, initializing [ 666.606539] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 666.610826] Lustre: Skipped 1 previous similar message [ 666.615228] Lustre: 26178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 666.728863] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 672.990735] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 675.000723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 675.015191] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 675.077049] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 677.544120] Lustre: 25163:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 677.555802] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 677.565308] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 677.574213] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 677.586805] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 677.592848] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 694.482592] Lustre: 26186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 694.493303] Lustre: 26186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 694.502361] Lustre: 26186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 694.508854] Lustre: 26186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 694.513364] Lustre: 26186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 694.517485] Lustre: 26186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.219151] Lustre: 25162:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 699.242173] Lustre: 25162:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 699.254186] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 699.261701] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.270625] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 699.279256] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.285273] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 699.290942] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.299994] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 699.304886] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 699.312745] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 699.322482] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.544432] Lustre: 26186:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 702.553497] Lustre: 26186:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 702.563947] Lustre: 26186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 702.569426] Lustre: 26186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.575941] Lustre: 26186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 702.587679] Lustre: 26186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.596459] Lustre: 26186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 702.604890] Lustre: 26186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.622242] Lustre: 26186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 702.629247] Lustre: 26186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 702.634730] Lustre: 26186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 702.643480] Lustre: 26186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.587619] Lustre: 25163:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 707.612064] Lustre: 25163:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message [ 707.627930] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.639424] Lustre: 25163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.647937] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.654320] Lustre: 25163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.661343] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 707.668435] Lustre: 25163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.674448] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.685401] Lustre: 25163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.693558] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.699805] Lustre: 25163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 718.384323] Lustre: 25162:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 718.400765] Lustre: 25162:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages [ 718.414110] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 718.419791] Lustre: 25162:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.426320] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 718.431218] Lustre: 25162:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.439036] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 718.446189] Lustre: 25162:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.457512] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 718.464547] Lustre: 25162:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 718.469643] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 718.475342] Lustre: 25162:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 766.947607] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.975051] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 772.067788] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 772.787500] Lustre: server umount lustre-OST07c6 complete [ 782.230245] Lustre: server umount lustre-MDT0000 complete [ 800.290579] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 803.337295] Key type lgssc unregistered [ 803.710511] LNet: 28735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.721432] LNetError: 28735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.744285] LNet: Removed LNI 192.168.206.102@tcp [ 804.764414] Key type .llcrypt unregistered [ 804.768397] Key type ._llcrypt unregistered [ 828.365705] Key type ._llcrypt registered [ 828.375623] Key type .llcrypt registered [ 828.466413] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 829.829890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.864095] alg: No test for adler32 (adler32-zlib) [ 831.002562] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 831.294695] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 832.999155] Key type lgssc registered [ 833.884083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 841.786619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 841.798731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.130133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 848.236564] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 852.454860] LustreError: 30011:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.481778] LustreError: 30011:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 856.720048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.193593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 862.012383] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 869.563828] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.830873] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 883.217696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 883.801541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 887.924829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 890.859989] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 898.932510] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.103847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 923.108511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.169308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 923.189743] Lustre: Skipped 1 previous similar message [ 928.224492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.346192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.351935] Lustre: Skipped 2 previous similar messages [ 934.879134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 935.161462] Lustre: server umount lustre-OST0000 complete [ 943.584190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.590702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.604674] Lustre: Skipped 2 previous similar messages [ 943.633564] Lustre: Skipped 1 previous similar message [ 945.436374] Lustre: server umount lustre-MDT0000 complete [ 948.711981] LustreError: 30012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.723268] LustreError: 30012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 949.088685] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128902 with bad export cookie 12371653395159307798 [ 949.108826] LustreError: 29991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 949.109602] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 949.553899] Lustre: server umount lustre-MDT0001 complete [ 961.280291] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 964.693240] Key type lgssc unregistered [ 965.034603] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.056942] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.123208] LNet: Removed LNI 192.168.206.102@tcp [ 966.978950] Key type .llcrypt unregistered [ 966.985595] Key type ._llcrypt unregistered [ 984.714752] Key type ._llcrypt registered [ 984.717370] Key type .llcrypt registered [ 984.816848] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 985.668544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 985.895670] alg: No test for adler32 (adler32-zlib) [ 986.899225] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 987.149640] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 988.871246] Key type lgssc registered [ 990.006788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 997.961658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 997.977888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.477566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1004.084077] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1009.644458] LustreError: 35028:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1009.660655] LustreError: 35028:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1012.593176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.123698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1017.693213] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1024.802979] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.505988] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1036.995789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.358907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.202524] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1044.548407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1051.140612] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1063.904173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.915349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.947770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.935650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1064.943719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.956194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1064.966766] Lustre: Skipped 1 previous similar message [ 1070.055729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1070.068714] Lustre: Skipped 1 previous similar message [ 1075.175479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1075.200571] Lustre: Skipped 1 previous similar message [ 1076.191401] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1076.425485] Lustre: server umount lustre-OST0000 complete [ 1080.289165] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1080.298990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1080.303092] Lustre: Skipped 1 previous similar message [ 1086.396900] Lustre: server umount lustre-MDT0000 complete [ 1090.264581] LustreError: 35010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129044 with bad export cookie 11951409167301297193 [ 1090.282329] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.688971] Lustre: server umount lustre-MDT0001 complete [ 1102.705623] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1106.219807] Key type lgssc unregistered [ 1106.605622] LNet: 38568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.611427] LNetError: 38568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.634165] LNet: Removed LNI 192.168.206.102@tcp [ 1107.818098] Key type .llcrypt unregistered [ 1107.820497] Key type ._llcrypt unregistered [ 1137.101642] Key type ._llcrypt registered [ 1137.114978] Key type .llcrypt registered [ 1137.271410] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1139.162859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.175804] alg: No test for adler32 (adler32-zlib) [ 1140.462587] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 1140.851470] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1142.592781] Key type lgssc registered [ 1143.828869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1151.802506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1151.850247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1153.346100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.120217] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1163.748130] LustreError: 39990:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.769093] LustreError: 39990:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1167.281608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.744793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1172.864644] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1179.698636] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1186.412249] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1193.419466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.746548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.910532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1199.446061] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1206.173066] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1218.531864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.540734] Lustre: Skipped 1 previous similar message [ 1218.548297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1218.554690] Lustre: Skipped 1 previous similar message [ 1219.341929] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 1223.024858] Lustre: server umount lustre-MDT0000 complete [ 1223.662294] LustreError: 39990:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.688832] LustreError: 39990:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1227.403833] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129181 with bad export cookie 3732049978836139301 [ 1227.410961] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1227.412074] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1228.768715] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.772613] LustreError: 39985:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1228.782023] Lustre: Skipped 1 previous similar message [ 1228.782967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1228.811994] LustreError: 39985:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1232.800220] LustreError: 40977:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1232.800964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1232.819364] LustreError: 40977:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1232.822839] Lustre: Skipped 1 previous similar message [ 1233.803611] Lustre: server umount lustre-MDT0001 complete [ 1249.305508] Lustre: server umount lustre-OST0000 complete [ 1266.433191] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1269.340601] Key type lgssc unregistered [ 1269.652104] LNet: 43852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.662336] LNetError: 43852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.678919] LNet: Removed LNI 192.168.206.102@tcp [ 1270.627585] Key type .llcrypt unregistered [ 1270.629593] Key type ._llcrypt unregistered [ 1294.298414] Key type ._llcrypt registered [ 1294.304473] Key type .llcrypt registered [ 1294.422290] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1295.671986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.829548] alg: No test for adler32 (adler32-zlib) [ 1297.021844] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 1297.378185] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1299.127551] Key type lgssc registered [ 1300.097508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1307.858498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1307.871708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.382483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.460947] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1319.912513] LustreError: 45271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1319.932574] LustreError: 45271:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1323.208498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1323.655383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1328.617262] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1336.037989] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1342.740432] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1349.448141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1349.610812] LustreError: 47088:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.895854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.017099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1356.652295] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1364.761722] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1370.604812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1370.617332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1370.631719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.911531] Lustre: lustre-OST0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 1375.712541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.718446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.848857] Lustre: server umount lustre-OST0000 complete [ 1394.562766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1394.568645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1394.577222] Lustre: Skipped 1 previous similar message [ 1394.595129] Lustre: Skipped 1 previous similar message [ 1400.809545] LustreError: 45265:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.824540] LustreError: 45265:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1400.916562] Lustre: server umount lustre-MDT0000 complete [ 1404.219855] LustreError: 45250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129357 with bad export cookie 11537781239022842705 [ 1404.224257] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.233191] LustreError: 45250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1410.605221] Lustre: server umount lustre-MDT0001 complete [ 1420.263879] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1422.941874] Key type lgssc unregistered [ 1423.233266] LNet: 48984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.243293] LNetError: 48984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.262517] LNet: Removed LNI 192.168.206.102@tcp [ 1424.200296] Key type .llcrypt unregistered [ 1424.202881] Key type ._llcrypt unregistered [ 1446.222620] Key type ._llcrypt registered [ 1446.224490] Key type .llcrypt registered [ 1446.346451] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1447.394363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1447.770334] alg: No test for adler32 (adler32-zlib) [ 1448.931344] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 1449.150458] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1450.799413] Key type lgssc registered [ 1451.892891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1459.045969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1459.072202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.466352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.597941] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1470.947617] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.962517] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1472.501037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1472.881752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1477.023320] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1484.267589] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1489.671190] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1496.118485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.478140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1502.574797] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1505.726221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1508.788178] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1513.832581] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1517.156893] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1529.338523] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-11 8h24m40s [ 1530.712788] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1534.433601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1534.435867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.445821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.464619] Lustre: Skipped 1 previous similar message [ 1535.532544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1535.539903] Lustre: Skipped 2 previous similar messages [ 1537.306212] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 1537.318028] Lustre: Skipped 3 previous similar messages [ 1538.938169] Lustre: server umount lustre-MDT0000 complete [ 1541.603159] LustreError: 51438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.632124] LustreError: 51438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1542.023860] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1547.755842] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.774422] LustreError: 50393:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1548.180424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.379889] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1548.506917] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (not set up) [ 1548.686982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.766685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1549.744247] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1553.271233] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1556.583541] Lustre: DEBUG MARKER: Wait for df (36755) ... [ 1557.978577] Lustre: DEBUG MARKER: done [ 1564.130891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.141776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.152158] Lustre: Skipped 1 previous similar message [ 1564.156221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1565.157201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.157985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.623650] Lustre: server umount lustre-OST0000 complete [ 1575.403593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1575.405707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1575.415204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.415210] Lustre: Skipped 2 previous similar messages [ 1575.445016] Lustre: Skipped 1 previous similar message [ 1578.791758] Lustre: server umount lustre-MDT0000 complete [ 1582.128463] LustreError: 50376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129535 with bad export cookie 2538467773963036278 [ 1582.161337] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.522280] Lustre: server umount lustre-MDT0001 complete [ 1593.634387] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1596.748093] Key type lgssc unregistered [ 1597.080065] LNet: 55315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.110769] LNetError: 55315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.144702] LNet: Removed LNI 192.168.206.102@tcp [ 1598.271807] Key type .llcrypt unregistered [ 1598.273848] Key type ._llcrypt unregistered [ 1637.043982] Key type ._llcrypt registered [ 1637.046493] Key type .llcrypt registered [ 1637.150962] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1638.322184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.354414] alg: No test for adler32 (adler32-zlib) [ 1639.331295] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 1639.531540] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1641.167182] Key type lgssc registered [ 1642.120510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1648.468514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1648.495401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1649.794969] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1649.809929] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1649.957680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.352501] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1661.825996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.900529] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1661.919728] Lustre: 57525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1662.102584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1666.018633] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1674.129705] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1679.261829] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1686.243321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.503171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1686.659148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.508027] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1695.922337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1705.317383] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1713.936218] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-11 8h27m44s [ 1715.641856] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1724.655132] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1727.348649] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1727.663245] LustreError: 57530:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1747.708756] Lustre: lustre-MDT0000: Client 07c38b62-8296-4fef-a089-cf9beb786dbd (at 192.168.206.2@tcp) reconnecting [ 1767.665815] LustreError: 57530:0:(mdt_handler.c:2352:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1769.733703] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1772.706571] Lustre: DEBUG MARKER: done [ 1782.760589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.770965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.789608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.264724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.284735] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1783.297775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1783.302850] Lustre: Skipped 1 previous similar message [ 1788.393181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.401918] Lustre: Skipped 1 previous similar message [ 1792.991371] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1793.152956] Lustre: server umount lustre-OST0000 complete [ 1798.625048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.649664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.659629] Lustre: Skipped 1 previous similar message [ 1802.720696] Lustre: server umount lustre-MDT0000 complete [ 1806.531655] LustreError: 57517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129760 with bad export cookie 2981966257717093608 [ 1806.536215] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.556680] LustreError: 57517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1807.024351] Lustre: server umount lustre-MDT0001 complete [ 1817.868766] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 1820.695241] Key type lgssc unregistered [ 1820.963727] LNet: 61867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.973375] LNetError: 61867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.999334] LNet: Removed LNI 192.168.206.102@tcp [ 1821.858562] Key type .llcrypt unregistered [ 1821.860456] Key type ._llcrypt unregistered [ 1866.867297] Key type ._llcrypt registered [ 1866.876884] Key type .llcrypt registered [ 1867.077718] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 1868.304191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1868.392731] alg: No test for adler32 (adler32-zlib) [ 1869.515715] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 1869.836691] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 1871.559173] Key type lgssc registered [ 1872.842951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.259988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.286298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.591157] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1877.615575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.781876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.701668] Lustre: Failing over lustre-MDT0000 [ 1879.056298] Lustre: server umount lustre-MDT0000 complete [ 1887.161291] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:30:38 (1789129838) [ 1895.563944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.115769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1902.130767] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1911.069109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.369674] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1911.435708] Lustre: 64668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1911.753135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1917.359612] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1924.921166] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.595168] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.243905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1937.400782] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1937.599990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1943.256417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1943.365624] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 1951.019345] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.633626] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1964.000061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.001313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1964.013019] Lustre: Skipped 2 previous similar messages [ 1964.037630] Lustre: Skipped 2 previous similar messages [ 1969.120773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.134120] Lustre: Skipped 2 previous similar messages [ 1974.241617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.267913] Lustre: Skipped 2 previous similar messages [ 1978.335159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1978.941317] Lustre: server umount lustre-MDT0000 complete [ 1983.955410] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789129937 with bad export cookie 12205788879211835301 [ 1983.963869] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.976750] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1984.510480] Lustre: server umount lustre-MDT0001 complete [ 1988.792526] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1993.748459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2000.868386] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129938/real 1789129938] req@00000000495e2d1e x1876038593323776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789129954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2000.915147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.285761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2004.959397] Lustre: 63596:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129942/real 1789129942] req@00000000fbb3fa0e x1876038593323904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789129958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2005.984238] Lustre: 63595:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129943/real 1789129943] req@00000000f4101125 x1876038593324032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789129959 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2009.652593] LustreError: 66517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.681788] LustreError: 66517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2009.793152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.959158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2010.719309] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129947/real 1789129947] req@000000007abbd08f x1876038593324288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789129963 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2010.786667] LustreError: 68437:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.795859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2010.810912] LustreError: 68437:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2014.879108] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789129952/real 1789129952] req@00000000a93ffea0 x1876038593324416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789129968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2015.817451] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2016.230534] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2016.265611] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2021.346137] LustreError: 66518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2021.379634] LustreError: 66518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2026.388569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.469552] LustreError: 68438:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2026.498818] LustreError: 68438:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2026.780258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2032.122601] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2032.306424] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2041.160736] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.235445] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2057.698638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2057.702982] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2057.709045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2057.730489] Lustre: Skipped 2 previous similar messages [ 2060.826704] Lustre: server umount lustre-MDT0000 complete [ 2062.831339] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2062.851932] LustreError: 68439:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2064.327370] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130018 with bad export cookie 12205788879211837576 [ 2064.333563] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.341468] LustreError: 64662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2064.661664] Lustre: server umount lustre-MDT0001 complete [ 2071.306771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2081.895742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.295293] Lustre: 63593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789130021/real 1789130021] req@00000000c8dd5e5b x1876038593352064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789130037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2083.347728] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2089.568968] LustreError: 63592:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000049041746 x1876038593353088/t0(0) o250->MGC192.168.206.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.074895] LustreError: 69406:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.160310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.244080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2095.283643] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2104.926502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.450918] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2111.041439] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2112.562614] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2120.697316] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.361048] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2136.033259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2136.056471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.078961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.084165] Lustre: Skipped 2 previous similar messages [ 2142.149735] Lustre: server umount lustre-MDT0000 complete [ 2143.216958] LustreError: 71125:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.234073] LustreError: 71125:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 2146.292465] LustreError: 67375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130100 with bad export cookie 12205788879211838955 [ 2146.292762] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2146.303869] LustreError: 67375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2146.818641] Lustre: server umount lustre-MDT0001 complete [ 2154.806081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2158.781302] Lustre: DEBUG MARKER: files compared the same [ 2171.170382] Lustre: server umount lustre-OST0000 complete [ 2187.506726] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2190.522567] Key type lgssc unregistered [ 2190.866130] LNet: 74406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.878515] LNetError: 74406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.895398] LNet: Removed LNI 192.168.206.102@tcp [ 2192.378426] Key type .llcrypt unregistered [ 2192.382536] Key type ._llcrypt unregistered [ 2220.065551] Key type ._llcrypt registered [ 2220.067807] Key type .llcrypt registered [ 2220.215599] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2221.285767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.422899] alg: No test for adler32 (adler32-zlib) [ 2222.679385] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 2223.077531] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2224.856082] Key type lgssc registered [ 2226.660890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2236.149037] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2236.186239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.935858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.010463] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug +malloc all [ 2248.164118] LustreError: 75822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.215671] LustreError: 75822:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2253.294658] LustreError: 75823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2257.832393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.152550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.860130] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug +malloc all [ 2273.626752] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.077222] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2287.635124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.043127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.178121] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug +malloc all [ 2301.381433] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2307.515952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2317.793519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2317.800756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.804046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2317.813398] Lustre: Skipped 1 previous similar message [ 2317.821880] Lustre: Skipped 1 previous similar message [ 2321.807493] Lustre: server umount lustre-OST0000 complete [ 2328.031827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.065112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.077740] Lustre: Skipped 1 previous similar message [ 2332.084978] Lustre: server umount lustre-MDT0000 complete [ 2333.158380] LustreError: 75823:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.182928] LustreError: 75823:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2336.272483] LustreError: 75802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130290 with bad export cookie 17473730426938636914 [ 2336.294719] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2336.922224] Lustre: server umount lustre-MDT0001 complete [ 2348.142543] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2350.982219] Key type lgssc unregistered [ 2351.248692] LNet: 79382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.255835] LNetError: 79382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.278196] LNet: Removed LNI 192.168.206.102@tcp [ 2351.997427] Key type .llcrypt unregistered [ 2351.999299] Key type ._llcrypt unregistered [ 2374.567109] Key type ._llcrypt registered [ 2374.575624] Key type .llcrypt registered [ 2374.685820] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2375.982413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.009933] alg: No test for adler32 (adler32-zlib) [ 2377.111800] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 2377.329809] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2379.023567] Key type lgssc registered [ 2380.036070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2386.640062] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.653508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.207498] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2407.328932] LustreError: 80771:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.102@tcp: failed processing log, type 4: rc = -110 [ 2434.015191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.200681] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2449.626514] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2455.820951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.891846] Lustre: *** cfs_fail_loc=706, val=0*** [ 2456.178491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.253895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2460.563278] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2461.674325] LustreError: 80797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.707863] LustreError: 80797:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2466.785952] LustreError: 80796:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.856178] LustreError: 80797:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.884112] LustreError: 80797:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2471.266785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.844485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2476.817488] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2485.785815] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.183772] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2496.992295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2496.997040] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.004658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.513596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2497.520286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.521655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.935912] Lustre: server umount lustre-OST0000 complete [ 2507.751063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.756047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2507.760500] Lustre: Skipped 1 previous similar message [ 2507.761512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2507.761518] Lustre: Skipped 1 previous similar message [ 2512.705758] Lustre: server umount lustre-MDT0000 complete [ 2512.870501] LustreError: 81857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.893813] LustreError: 81857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2517.486223] LustreError: 80779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130471 with bad export cookie 16751724993601696082 [ 2517.489851] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2517.515744] LustreError: 80779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2517.919664] Lustre: server umount lustre-MDT0001 complete [ 2531.963928] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2534.833274] Key type lgssc unregistered [ 2535.149431] LNet: 84393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.161932] LNetError: 84393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.177145] LNet: Removed LNI 192.168.206.102@tcp [ 2536.005940] Key type .llcrypt unregistered [ 2536.013892] Key type ._llcrypt unregistered [ 2559.086600] Key type ._llcrypt registered [ 2559.090549] Key type .llcrypt registered [ 2559.184550] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2559.970100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2559.982556] alg: No test for adler32 (adler32-zlib) [ 2561.020241] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 2561.316176] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2562.999291] Key type lgssc registered [ 2564.350574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.251345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2572.296696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2578.152872] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2585.052509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.514734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2590.024221] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2590.690806] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.714993] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2595.809944] LustreError: 86501:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.827477] LustreError: 86501:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2598.271241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2598.644760] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.677628] LustreError: 86498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2598.695457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2604.002089] LustreError: 86502:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.018740] LustreError: 86502:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2604.997500] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2611.107909] LustreError: 86501:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.148027] LustreError: 86501:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2613.756273] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2613.774644] Lustre: MGS: Client 0bbff174-b96c-44df-9fdc-81eca70a2c38 (at 0@lo) reconnecting [ 2613.815124] LustreError: 86512:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2614.042486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2614.212530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2619.190806] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2634.724657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.725935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.731874] LustreError: Skipped 1 previous similar message [ 2634.732291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.778034] Lustre: Skipped 1 previous similar message [ 2638.331962] Lustre: server umount lustre-OST0000 complete [ 2644.969772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2644.983216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.993842] Lustre: Skipped 2 previous similar messages [ 2648.289496] Lustre: server umount lustre-MDT0000 complete [ 2650.082871] LustreError: 86496:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2650.128951] LustreError: 86496:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2653.095436] Lustre: server umount lustre-MDT0001 complete [ 2657.466937] Lustre: server umount MGS complete [ 2669.440396] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:43:40 (1789130620) [ 2700.327735] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 2710.666849] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2748.471855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.216288] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2765.336386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.549352] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2765.609630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2765.719917] Lustre: lustre-MDT0001: new disk, initializing [ 2765.826618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2765.877327] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2765.890243] Lustre: 93669:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2765.897270] LustreError: 93669:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2770.397553] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2780.912610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2781.235908] Lustre: lustre-OST0000: new disk, initializing [ 2781.242454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2781.249092] Lustre: 94436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.338026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2788.268464] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2790.498517] Lustre: 94849:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000076493b1 x1876039318938112/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789130760 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2797.395951] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2804.146832] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.157104] Lustre: MGS: Client d98a1a84-a9fe-45bc-89d3-7874cf99e96a (at 0@lo) reconnecting [ 2804.202249] Lustre: 95481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2804.217229] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2804.234856] LustreError: 93655:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2804.257451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2804.354505] Lustre: lustre-MDT0000: new disk, initializing [ 2804.529153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.571792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2804.713191] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2805.665260] Lustre: 95505:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000077bc046f x1876039318944768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789130775 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2809.550250] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2810.877677] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2814.765918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2814.778780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2814.862391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2840.034181] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.039282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2840.044213] Lustre: Skipped 2 previous similar messages [ 2840.062839] Lustre: Skipped 1 previous similar message [ 2845.159589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.168063] Lustre: Skipped 1 previous similar message [ 2845.553549] Lustre: server umount lustre-OST0000 complete [ 2850.278050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2850.279153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2850.301852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.313567] Lustre: Skipped 1 previous similar message [ 2850.322858] Lustre: Skipped 1 previous similar message [ 2855.638056] Lustre: server umount lustre-MDT0000 complete [ 2866.094481] Lustre: server umount lustre-MDT0001 complete [ 2870.077945] Lustre: server umount MGS complete [ 2881.963539] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:47:13 (1789130833) [ 2900.933414] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 2904.141406] Key type lgssc unregistered [ 2904.449745] LNet: 98009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2904.462594] LNetError: 98009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2904.482386] LNet: Removed LNI 192.168.206.102@tcp [ 2905.370701] Key type .llcrypt unregistered [ 2905.375668] Key type ._llcrypt unregistered [ 2922.743084] Key type ._llcrypt registered [ 2922.746805] Key type .llcrypt registered [ 2922.821638] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 2923.787402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2923.828699] alg: No test for adler32 (adler32-zlib) [ 2924.964789] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 2925.244912] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 2926.944457] Key type lgssc registered [ 2928.048937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2936.146425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2936.206716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.709377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.488510] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2948.067342] LustreError: 99231:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.113514] LustreError: 99231:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2953.197194] LustreError: 99232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.589914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.082048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2959.398672] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2968.089366] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2974.245399] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2981.531562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2981.925920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.970565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2988.542399] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 2995.851742] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3008.481340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3008.487203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3008.528221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.509054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.512787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.523432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.723246] Lustre: server umount lustre-OST0000 complete [ 3016.672227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3016.675486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3016.694855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3016.704074] Lustre: Skipped 1 previous similar message [ 3019.747852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3019.756079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.676428] Lustre: server umount lustre-MDT0000 complete [ 3024.868928] LustreError: 99227:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.891209] LustreError: 99227:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3026.846966] LustreError: 99211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130980 with bad export cookie 16465104985298152727 [ 3026.852587] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3026.862260] LustreError: 99211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3027.292299] Lustre: server umount lustre-MDT0001 complete [ 3038.352724] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3041.303071] Key type lgssc unregistered [ 3041.611911] LNet: 102767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3041.621279] LNetError: 102767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3042.665227] LNet: Removed LNI 192.168.206.102@tcp [ 3043.769951] Key type .llcrypt unregistered [ 3043.780523] Key type ._llcrypt unregistered [ 3060.037736] Key type ._llcrypt registered [ 3060.040601] Key type .llcrypt registered [ 3060.137864] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3061.281962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3061.299650] alg: No test for adler32 (adler32-zlib) [ 3062.422239] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3062.703055] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3064.383185] Key type lgssc registered [ 3065.657800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3072.630565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3072.640491] LustreError: 103838:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3072.728936] LustreError: 103852:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3072.734767] LustreError: 103838:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4911 [ 3072.747171] LustreError: 103838:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3072.752629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.755186] LustreError: 103838:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3074.301563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.096592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.325698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.609166] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3091.636409] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing lsmod [ 3094.822335] LustreError: 105049:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3094.875567] LustreError: 105059:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3094.893554] LustreError: 105049:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4934 [ 3094.903865] LustreError: 105049:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3094.912655] LustreError: 105049:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3094.934306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.328626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3100.651073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.659522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3100.669496] Lustre: Skipped 1 previous similar message [ 3104.564534] Lustre: server umount lustre-MDT0000 complete [ 3105.761001] LustreError: 103890:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3105.784500] LustreError: 103890:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3107.810379] LustreError: 103886:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3107.842698] LustreError: 103886:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3108.664239] LustreError: 103864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131062 with bad export cookie 18130923616685733771 [ 3108.690393] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3108.704084] LustreError: 103864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3109.919268] LustreError: 104321:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c705df93 x1876039844253440/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3110.227269] Lustre: server umount lustre-MDT0001 complete [ 3125.083779] Lustre: server umount lustre-OST0000 complete [ 3135.079923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3135.695947] LustreError: 106122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3135.809354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.908574] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3141.089885] LustreError: 106123:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.217200] LustreError: 106122:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3148.043918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.325364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3152.453281] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3159.602285] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.219204] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3171.105704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3171.507423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.508494] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3180.733071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3183.127550] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3189.217996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.231969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.242034] Lustre: Skipped 1 previous similar message [ 3189.249706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3189.254129] Lustre: Skipped 2 previous similar messages [ 3191.268282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3191.269962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3191.274687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.570193] Lustre: server umount lustre-OST0000 complete [ 3199.460198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3199.469379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.486615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3199.492907] Lustre: Skipped 1 previous similar message [ 3206.634121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.640330] Lustre: Skipped 3 previous similar messages [ 3212.255370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3212.563443] Lustre: server umount lustre-MDT0000 complete [ 3216.635756] LustreError: 106103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131170 with bad export cookie 18130923616685734471 [ 3216.639690] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3216.644788] LustreError: 106103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3217.274406] Lustre: server umount lustre-MDT0001 complete [ 3228.417873] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3231.296452] Key type lgssc unregistered [ 3231.623756] LNet: 109662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.629385] LNetError: 109662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.674433] LNet: Removed LNI 192.168.206.102@tcp [ 3232.459681] Key type .llcrypt unregistered [ 3232.462338] Key type ._llcrypt unregistered [ 3256.309731] Key type ._llcrypt registered [ 3256.315116] Key type .llcrypt registered [ 3256.430443] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3257.176638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3257.296907] alg: No test for adler32 (adler32-zlib) [ 3258.375607] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3258.629762] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3260.287221] Key type lgssc registered [ 3261.258966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3268.481622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3268.512079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.879957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.094937] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3279.338086] LustreError: 111079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.356481] LustreError: 111079:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3281.861198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.253709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.627492] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3294.065226] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3299.617439] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3306.467391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3306.923652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3312.994206] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3315.129041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3320.125264] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3325.822924] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3328.851438] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3333.601671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3333.616058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.633332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.650847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3335.658964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3335.664356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.111571] Lustre: server umount lustre-OST0000 complete [ 3343.840412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3343.866540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.908176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.920411] Lustre: Skipped 1 previous similar message [ 3345.907954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3345.930233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3345.937897] Lustre: Skipped 1 previous similar message [ 3349.592731] Lustre: server umount lustre-MDT0000 complete [ 3351.019759] LustreError: 111080:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3351.054925] LustreError: 111080:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3353.689297] LustreError: 111060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131307 with bad export cookie 17345670404760031618 [ 3353.704324] LustreError: 111060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3353.707289] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3354.017376] Lustre: server umount lustre-MDT0001 complete [ 3366.168349] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3368.652668] Key type lgssc unregistered [ 3369.077154] LNet: 114718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3369.095488] LNetError: 114718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3369.114802] LNet: Removed LNI 192.168.206.102@tcp [ 3370.103575] Key type .llcrypt unregistered [ 3370.106139] Key type ._llcrypt unregistered [ 3384.686030] Key type ._llcrypt registered [ 3384.690184] Key type .llcrypt registered [ 3384.815879] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3385.909907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3385.959488] alg: No test for adler32 (adler32-zlib) [ 3386.992771] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3387.250664] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3388.951159] Key type lgssc registered [ 3390.213475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3398.311882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3398.333882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3399.745184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.386716] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3409.902143] LustreError: 115917:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.934351] LustreError: 115917:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3413.552096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3413.935328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3418.046849] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3425.194598] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3430.348222] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3437.288385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3437.695772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3443.429520] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3445.948412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3449.937877] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3460.576485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3460.590660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.606299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.604135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3461.609580] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3461.628260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.113839] Lustre: server umount lustre-OST0000 complete [ 3470.304857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3470.309776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.321718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3470.325402] Lustre: Skipped 1 previous similar message [ 3473.432298] Lustre: server umount lustre-MDT0000 complete [ 3476.960906] LustreError: 115912:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3476.982104] LustreError: 115912:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3477.778800] LustreError: 115897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131431 with bad export cookie 14997630821526747423 [ 3477.792327] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3478.426791] Lustre: server umount lustre-MDT0001 complete [ 3488.814188] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3491.454348] Key type lgssc unregistered [ 3491.681004] LNet: 119453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3491.689599] LNetError: 119453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3491.716972] LNet: Removed LNI 192.168.206.102@tcp [ 3492.394247] Key type .llcrypt unregistered [ 3492.396653] Key type ._llcrypt unregistered [ 3513.412613] Key type ._llcrypt registered [ 3513.415719] Key type .llcrypt registered [ 3513.507257] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3514.672727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3514.707073] alg: No test for adler32 (adler32-zlib) [ 3515.845541] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3516.109487] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3517.807652] Key type lgssc registered [ 3518.988788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3525.528523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3525.549524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.983961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.735541] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3536.366767] LustreError: 120854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.382763] LustreError: 120854:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3540.590874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3540.826613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3544.497588] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3551.320904] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3557.325987] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3563.986725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3564.292988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.917541] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3572.542033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3576.899190] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3584.838973] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3588.184957] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3588.196259] Lustre: Skipped 2 previous similar messages [ 3591.302931] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3591.338587] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3592.039896] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3592.048207] Lustre: Skipped 2 previous similar messages [ 3597.886218] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3597.901269] Lustre: Skipped 1 previous similar message [ 3602.447500] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.2@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3602.456126] Lustre: Skipped 2 previous similar messages [ 3604.013538] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.2@tcp 192.168.0.[2,10]@tcp [ 3613.893747] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.2/24@tcp in log lustre-MDT0000 [ 3613.902784] Lustre: Skipped 2 previous similar messages [ 3623.484148] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.2/24@tcp [ 3623.488792] Lustre: Skipped 1 previous similar message [ 3629.034468] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3629.034671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3629.056931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3629.066519] Lustre: Skipped 1 previous similar message [ 3634.030864] Lustre: server umount lustre-OST0000 complete [ 3638.240314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3638.250809] LustreError: Skipped 1 previous similar message [ 3638.257979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3638.286670] Lustre: Skipped 1 previous similar message [ 3638.308316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3639.266911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3643.597250] Lustre: server umount lustre-MDT0000 complete [ 3644.388042] LustreError: 120854:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.060835] LustreError: 120835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131600 with bad export cookie 7087864741068155733 [ 3647.064081] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3647.071791] LustreError: 120835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3647.456740] Lustre: server umount lustre-MDT0001 complete [ 3657.213151] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3659.826898] Key type lgssc unregistered [ 3660.115645] LNet: 125848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3660.123729] LNetError: 125848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3660.145525] LNet: Removed LNI 192.168.206.102@tcp [ 3661.166059] Key type .llcrypt unregistered [ 3661.168267] Key type ._llcrypt unregistered [ 3685.270073] Key type ._llcrypt registered [ 3685.272019] Key type .llcrypt registered [ 3685.350331] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3686.287353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3686.302267] alg: No test for adler32 (adler32-zlib) [ 3687.339709] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3687.613324] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3689.287928] Key type lgssc registered [ 3690.458740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3697.275373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3697.300512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3698.728206] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3698.754822] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.2/24@tcp [ 3698.766310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.027826] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3708.897813] LustreError: 127388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3708.929360] LustreError: 127388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3710.775869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.185981] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3711.209870] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.2/24@tcp [ 3711.249294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3715.701316] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3723.260577] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.971564] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3734.235929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3734.609300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.038388] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3743.859222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3746.780685] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3756.512218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3756.519839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3756.534497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.027808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3757.040366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.054940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.584617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3759.593105] Lustre: Skipped 1 previous similar message [ 3762.053874] Lustre: server umount lustre-OST0000 complete [ 3767.783467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3767.791583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.798544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.834555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.949083] Lustre: server umount lustre-MDT0000 complete [ 3774.944162] LustreError: 127388:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3774.958179] LustreError: 127388:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3775.871055] LustreError: 127368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131729 with bad export cookie 16024264735470474315 [ 3775.876504] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.885675] LustreError: 127368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3776.171922] Lustre: server umount lustre-MDT0001 complete [ 3788.734100] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 3792.814792] Key type lgssc unregistered [ 3793.217833] LNet: 130987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3793.233579] LNetError: 130987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3794.282797] LNet: Removed LNI 192.168.206.102@tcp [ 3795.338650] Key type .llcrypt unregistered [ 3795.341066] Key type ._llcrypt unregistered [ 3823.170584] Key type ._llcrypt registered [ 3823.173936] Key type .llcrypt registered [ 3823.280171] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 3838.529528] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 3839.653147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3839.683335] alg: No test for adler32 (adler32-zlib) [ 3840.731686] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 3841.039579] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 3842.743572] Key type lgssc registered [ 3843.827990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.022411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3876.042793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3877.357171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3877.393623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3877.509494] Lustre: lustre-MDT0000: new disk, initializing [ 3877.598125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.623603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3882.323597] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3893.695036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3893.840573] Lustre: 134390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3893.911196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3893.917926] Lustre: Skipped 1 previous similar message [ 3894.083655] Lustre: lustre-MDT0001: new disk, initializing [ 3894.208280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.283603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3894.313155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3900.599110] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3910.509427] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3916.440656] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3923.135366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3923.363628] Lustre: lustre-OST0000: new disk, initializing [ 3923.368180] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3923.377838] Lustre: 136359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3923.494335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.647137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3928.661581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3928.795822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3929.442333] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3939.221755] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3943.276942] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3943.436286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3947.081176] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3947.257768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3949.539957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3949.540691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.545803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.548226] LustreError: Skipped 1 previous similar message [ 3949.572891] Lustre: Skipped 1 previous similar message [ 3954.663119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3954.667589] Lustre: Skipped 1 previous similar message [ 3959.780576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3959.789134] Lustre: Skipped 1 previous similar message [ 3963.359192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3963.501818] Lustre: server umount lustre-OST0000 complete [ 3970.020282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.022027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3970.039018] Lustre: Skipped 1 previous similar message [ 3970.046665] Lustre: Skipped 1 previous similar message [ 3972.629821] Lustre: server umount lustre-MDT0000 complete [ 3975.386952] LustreError: 134383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131929 with bad export cookie 17507308056526043294 [ 3975.391446] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3975.403363] LustreError: 134383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3975.756354] Lustre: server umount lustre-MDT0001 complete [ 3983.101582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.443956] LustreError: 138578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3983.462721] LustreError: 138578:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 3983.509239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3987.511883] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 3988.963098] LustreError: 138579:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.081857] LustreError: 138578:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3996.268295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.660995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4000.577885] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4008.252912] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4014.245337] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4022.068198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4022.524696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.620620] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4038.425548] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4042.586250] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4042.760320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4046.955489] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4047.223918] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4055.430884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4055.598412] Lustre: lustre-OST0001: new disk, initializing [ 4055.602240] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4055.606873] Lustre: 141949:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4055.690161] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.111700] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4064.867415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4064.898384] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4065.059697] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4070.258503] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4070.511774] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4074.832995] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4075.077082] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4081.104657] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff943812f94000.ost_server_uuid 50 [ 4082.894666] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff943812f94000.ost_server_uuid in FULL state after 0 sec [ 4093.139210] Lustre: server umount lustre-OST0001 complete [ 4095.976214] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.980513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4095.994551] LustreError: 142414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.000458] Lustre: Skipped 1 previous similar message [ 4096.023384] LustreError: Skipped 1 previous similar message [ 4098.528000] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4098.538204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4098.551751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4098.559104] Lustre: Skipped 1 previous similar message [ 4100.577056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4101.095359] LustreError: 140399:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.111893] LustreError: 140399:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4111.327552] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4111.346853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4111.356171] LustreError: 140398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.360029] Lustre: Skipped 6 previous similar messages [ 4111.395607] LustreError: 140398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4111.633277] Lustre: server umount lustre-OST0000 complete [ 4116.452511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.470721] Lustre: Skipped 2 previous similar messages [ 4121.663209] Lustre: server umount lustre-MDT0000 complete [ 4125.130640] LustreError: 138559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132078 with bad export cookie 17507308056526044176 [ 4125.137678] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.149244] LustreError: 138559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4125.534665] Lustre: server umount lustre-MDT0001 complete [ 4144.053227] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 4146.878488] Key type lgssc unregistered [ 4147.154852] LNet: 144956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4147.168577] LNetError: 144956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4147.198868] LNet: Removed LNI 192.168.206.102@tcp [ 4148.030421] Key type .llcrypt unregistered [ 4148.034818] Key type ._llcrypt unregistered [ 4185.099068] Key type ._llcrypt registered [ 4185.100967] Key type .llcrypt registered [ 4185.252624] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 4200.714931] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 4202.635289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4202.667840] alg: No test for adler32 (adler32-zlib) [ 4204.041883] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 4204.407569] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 4206.087499] Key type lgssc registered [ 4207.291197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4241.143579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4241.189364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.567215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4242.637386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4242.738308] Lustre: lustre-MDT0000: new disk, initializing [ 4242.801187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.827468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4247.076946] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4256.661926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.785296] Lustre: 149116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4256.882067] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4256.889772] Lustre: Skipped 1 previous similar message [ 4256.982872] Lustre: lustre-MDT0001: new disk, initializing [ 4257.096263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4257.160195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4257.173853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4261.347622] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4270.536549] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4276.832739] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4283.803708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.181208] Lustre: lustre-OST0000: new disk, initializing [ 4284.186511] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4284.193658] Lustre: 151081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4284.290207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.596947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4289.610354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4289.740872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4289.789207] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4299.406465] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.359910] Lustre: Failing over lustre-OST0000 [ 4304.544664] Lustre: server umount lustre-OST0000 complete [ 4321.759919] Lustre: 147338:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132259/real 1789132259] req@00000000984cc996 x1876041041740928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789132275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4321.760248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.778617] Lustre: 147338:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4322.784134] Lustre: 147336:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132260/real 1789132260] req@00000000cd15e25a x1876041041741184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789132276 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4322.977541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4323.333285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.355460] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4323.585665] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4326.367440] Lustre: 147339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132264/real 1789132264] req@000000007fa54a46 x1876041041742336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789132280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4326.399954] Lustre: 147339:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4329.949139] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4331.487173] Lustre: 147338:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132269/real 1789132269] req@000000003e5d4feb x1876041041743360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789132285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4331.563323] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4331.566182] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4333.777979] Lustre: Failing over lustre-MDT0000 [ 4333.820318] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 4334.064740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.075549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.090609] Lustre: Skipped 1 previous similar message [ 4334.225950] Lustre: server umount lustre-MDT0000 complete [ 4336.615721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4336.631611] Lustre: Skipped 1 previous similar message [ 4337.631212] Lustre: 147339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132274/real 1789132274] req@000000007659698f x1876041041744512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789132290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4337.661209] Lustre: 147339:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4341.732601] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4341.757613] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4344.033845] LustreError: 149121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.857707] LustreError: 150778:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.886837] LustreError: 150778:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4349.153621] LustreError: 149121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.024529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.055193] Lustre: 147339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789132290/real 1789132290] req@00000000d0569cf4 x1876041041754496/t0(0) o400->MGC192.168.206.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789132306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4353.077220] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4354.101255] LustreError: 149121:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4354.131598] LustreError: 149121:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4363.235862] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.257900] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 4363.467910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4363.522264] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4364.504936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4367.215486] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4368.891869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4368.897633] Lustre: Skipped 1 previous similar message [ 4368.916183] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4368.967317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4377.569721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4377.579721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.610449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.113788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4379.129236] Lustre: Skipped 1 previous similar message [ 4381.579421] Lustre: server umount lustre-OST0000 complete [ 4389.349175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4389.350894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4389.357446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4389.381148] Lustre: Skipped 2 previous similar messages [ 4389.412163] Lustre: Skipped 1 previous similar message [ 4391.650970] Lustre: server umount lustre-MDT0000 complete [ 4394.465094] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.489498] LustreError: 149127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4395.763058] LustreError: 149107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132349 with bad export cookie 6491813308230449576 [ 4395.774981] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.784289] LustreError: 149107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4396.356773] Lustre: server umount lustre-MDT0001 complete [ 4406.562847] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 4409.704397] Key type lgssc unregistered [ 4409.986978] LNet: 154740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4409.999028] LNetError: 154740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4410.021198] LNet: Removed LNI 192.168.206.102@tcp [ 4410.910374] Key type .llcrypt unregistered [ 4410.913090] Key type ._llcrypt unregistered [ 4446.641318] Key type ._llcrypt registered [ 4446.645166] Key type .llcrypt registered [ 4446.731328] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 4447.743861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4447.796147] alg: No test for adler32 (adler32-zlib) [ 4448.916588] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 4449.187226] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 4450.913509] Key type lgssc registered [ 4452.110639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4459.148557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4459.161170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.690717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4464.843680] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4470.244596] LustreError: 156265:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.269258] LustreError: 156265:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4473.797624] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4475.366868] LustreError: 156266:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.245041] LustreError: 156294:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4478.255863] LustreError: 156294:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4478.265687] LustreError: 156294:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4478.643423] Lustre: server umount lustre-MDT0000 complete [ 4493.203977] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 4504.467124] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 4539.278288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.610814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4539.647631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4539.760134] Lustre: lustre-MDT0000: new disk, initializing [ 4539.876580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.902746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4544.842879] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4554.655500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4554.755518] Lustre: 159598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4554.802828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4554.806354] Lustre: Skipped 1 previous similar message [ 4554.874557] Lustre: lustre-MDT0001: new disk, initializing [ 4554.964877] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4555.010981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4555.021152] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4558.368539] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4567.154898] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4573.134813] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4582.637051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.003728] Lustre: lustre-OST0000: new disk, initializing [ 4583.014154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4583.025629] Lustre: 161564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4583.142813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.592578] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4591.666465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4591.679771] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4591.776490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4597.819730] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4602.066721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4603.777909] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:15:55 (1789132555) [ 4817.376835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.380374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.395694] Lustre: Skipped 1 previous similar message [ 4817.412493] Lustre: Skipped 2 previous similar messages [ 4822.496876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4822.503616] Lustre: Skipped 2 previous similar messages [ 4827.640530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4827.647868] Lustre: Skipped 1 previous similar message [ 4831.199188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4831.396323] Lustre: server umount lustre-MDT0000 complete [ 4832.737140] LustreError: 159604:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.768386] LustreError: 159604:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4834.981822] LustreError: 159588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132788 with bad export cookie 14339560270611151888 [ 4834.981963] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4834.990933] LustreError: 159588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4837.856275] LustreError: 161572:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.856539] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.896771] Lustre: Skipped 1 previous similar message [ 4837.909970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4841.364288] Lustre: server umount lustre-MDT0001 complete [ 4851.877623] Lustre: server umount lustre-OST0000 complete [ 4859.746182] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 4868.707680] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 4901.004303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.192685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4901.214374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4901.273475] Lustre: lustre-MDT0000: new disk, initializing [ 4901.325879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4901.343231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4905.273614] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4915.010562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.165650] Lustre: 166220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4915.230694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4915.235079] Lustre: Skipped 1 previous similar message [ 4915.291906] Lustre: lustre-MDT0001: new disk, initializing [ 4915.354204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4915.393238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4915.402673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4919.469050] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4929.591206] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4936.366435] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4943.055660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4943.372315] Lustre: lustre-OST0000: new disk, initializing [ 4943.377435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4943.383330] Lustre: 168191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4943.471359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4949.165699] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 4953.648830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4953.662375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4953.746070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4958.175372] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4962.723622] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4962.986992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4967.898400] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4968.081792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4974.563165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.572928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4974.581117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4974.581128] Lustre: Skipped 1 previous similar message [ 4974.584153] Lustre: Skipped 1 previous similar message [ 4984.287197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4984.654060] Lustre: server umount lustre-OST0000 complete [ 4989.920721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4989.922599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4989.950677] Lustre: Skipped 1 previous similar message [ 4989.963450] Lustre: Skipped 4 previous similar messages [ 4994.706566] Lustre: server umount lustre-MDT0000 complete [ 4995.040428] LustreError: 166232:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.074961] LustreError: 166232:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 4998.959282] LustreError: 167025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132952 with bad export cookie 14339560270611310718 [ 4998.963558] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4998.971736] LustreError: 167025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4999.275988] Lustre: server umount lustre-MDT0001 complete [ 5007.250840] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:22:38 (1789132958) [ 5024.907822] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 5034.625693] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 5069.595695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.892496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5069.950653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5070.051257] Lustre: lustre-MDT0000: new disk, initializing [ 5070.186282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.215881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5075.145575] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5086.087939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5086.311177] Lustre: 172946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5086.650321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5091.176259] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5100.215473] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5105.822224] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5113.255577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5113.567960] Lustre: lustre-OST0000: new disk, initializing [ 5113.577486] Lustre: Skipped 1 previous similar message [ 5113.585073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5113.593956] Lustre: Skipped 2 previous similar messages [ 5113.599487] Lustre: 174923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5119.564054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5119.578321] Lustre: Skipped 1 previous similar message [ 5119.584582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5119.714057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5121.361492] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5131.741575] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5140.459105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5140.459204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.460945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5140.469573] LustreError: Skipped 1 previous similar message [ 5140.495333] Lustre: Skipped 1 previous similar message [ 5144.816278] Lustre: server umount lustre-OST0000 complete [ 5150.692151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.711434] Lustre: Skipped 1 previous similar message [ 5154.755905] Lustre: server umount lustre-MDT0000 complete [ 5155.808205] LustreError: 172954:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5155.839865] LustreError: 172954:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5158.358476] LustreError: 172939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133112 with bad export cookie 14339560270611311600 [ 5158.369550] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5158.375401] LustreError: 172939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5158.734601] Lustre: server umount lustre-MDT0001 complete [ 5166.505968] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:25:18 (1789133118) [ 5181.536311] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 5191.504749] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 5223.844720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.082278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5224.119066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5224.197834] Lustre: lustre-MDT0000: new disk, initializing [ 5224.272119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5224.276158] Lustre: Skipped 2 previous similar messages [ 5224.286861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5228.671748] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5239.450089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.617204] Lustre: 179246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5239.867703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5244.411656] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5253.608919] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5258.654832] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5265.609354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5265.855389] Lustre: 181224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5271.681776] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5275.711657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5275.853510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5280.385459] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5291.490243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.490666] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5291.496717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5291.499407] LustreError: Skipped 1 previous similar message [ 5291.501972] Lustre: Skipped 4 previous similar messages [ 5293.515349] Lustre: server umount lustre-OST0000 complete [ 5301.223092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5304.091684] Lustre: server umount lustre-MDT0000 complete [ 5306.849361] LustreError: 180415:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5306.874704] LustreError: 180415:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5308.737635] LustreError: 180055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133262 with bad export cookie 14339560270611312678 [ 5308.746502] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5308.750700] LustreError: 180055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5309.249763] Lustre: server umount lustre-MDT0001 complete [ 5318.204498] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 5320.401330] Key type lgssc unregistered [ 5320.621626] LNet: 183083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.627995] LNetError: 183083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5320.641716] LNet: Removed LNI 192.168.206.102@tcp [ 5321.267435] Key type .llcrypt unregistered [ 5321.271108] Key type ._llcrypt unregistered [ 5340.658813] Key type ._llcrypt registered [ 5340.661441] Key type .llcrypt registered [ 5340.749693] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 5341.677210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5341.744285] alg: No test for adler32 (adler32-zlib) [ 5342.780434] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 5343.018929] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 5344.695178] Key type lgssc registered [ 5345.764244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5353.286559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5353.309238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5354.841184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.126314] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5365.230844] LustreError: 184479:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.259202] LustreError: 184479:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5366.547312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.923224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.021640] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5377.882446] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5383.413534] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5389.954190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5390.307841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5396.642504] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5400.578820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5405.047936] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5426.148417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.152240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5426.175499] Lustre: Skipped 1 previous similar message [ 5426.196813] Lustre: Skipped 1 previous similar message [ 5428.661148] Lustre: server umount lustre-OST0000 complete [ 5433.314505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5433.321855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.336666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.385182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5436.388038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.794536] Lustre: server umount lustre-MDT0000 complete [ 5441.509862] LustreError: 184480:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5441.536416] LustreError: 184480:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5443.100336] LustreError: 184459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133396 with bad export cookie 10631136176452197178 [ 5443.106889] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5443.108857] LustreError: 184459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5443.532859] Lustre: server umount lustre-MDT0001 complete [ 5455.675560] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 5459.052134] Key type lgssc unregistered [ 5459.474889] LNet: 188038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5459.493342] LNetError: 188038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5459.520493] LNet: Removed LNI 192.168.206.102@tcp [ 5460.861235] Key type .llcrypt unregistered [ 5460.863076] Key type ._llcrypt unregistered [ 5488.744257] Key type ._llcrypt registered [ 5488.748877] Key type .llcrypt registered [ 5488.909900] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 5490.951968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5491.068342] alg: No test for adler32 (adler32-zlib) [ 5492.172474] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 5492.500277] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 5494.191152] Key type lgssc registered [ 5495.676247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5504.708906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5504.723765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.242381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5511.616881] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5515.751186] LustreError: 189454:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.763796] LustreError: 189454:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5520.748685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5520.866898] LustreError: 189455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5521.405624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5527.986835] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5537.742105] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5543.999212] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5553.204621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5553.589202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5559.586829] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5566.753311] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5576.115356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5586.401842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.401842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.402535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5586.436137] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.489982] Lustre: Skipped 1 previous similar message [ 5587.728724] Lustre: lustre-OST0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 5589.991062] Lustre: server umount lustre-OST0000 complete [ 5599.077917] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff943804d4d000.ost_server_uuid 50 [ 5601.000423] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5617.126981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5617.128899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5617.132081] Lustre: Skipped 1 previous similar message [ 5617.145278] Lustre: Skipped 1 previous similar message [ 5621.769601] Lustre: server umount lustre-MDT0000 complete [ 5622.240705] LustreError: 192323:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5622.259779] LustreError: 192323:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5625.730931] LustreError: 189434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133579 with bad export cookie 14063990525752795462 [ 5625.740573] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5626.061345] Lustre: server umount lustre-MDT0001 complete [ 5638.433346] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:33:09 (1789133589) [ 5647.563897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5648.140326] LustreError: 193396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5648.308077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.478228] LustreError: 193397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.106551] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5662.695169] LustreError: 193397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.704223] LustreError: 193397:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5663.800081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.366630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5670.081820] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5678.190771] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5683.842506] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5690.742497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.150212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.235708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5697.790752] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5706.233456] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5714.220448] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5714.499719] Lustre: lustre-OST0001: new disk, initializing [ 5714.503986] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5714.518323] Lustre: 196237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5714.629334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5722.181168] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5722.324902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5722.339240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5722.533369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5734.711826] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.954286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.976134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5737.985699] Lustre: Skipped 1 previous similar message [ 5743.085019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.104265] Lustre: Skipped 3 previous similar messages [ 5743.721744] Lustre: server umount lustre-OST0000 complete [ 5745.376619] LustreError: 195217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.691489] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5750.932771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5755.669212] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5755.940965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5773.805884] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5773.810826] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5773.822603] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5773.827818] Lustre: Skipped 1 previous similar message [ 5776.550884] Lustre: server umount lustre-OST0001 complete [ 5781.984927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5781.990017] LustreError: Skipped 1 previous similar message [ 5781.997823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5782.005857] Lustre: Skipped 1 previous similar message [ 5786.977876] Lustre: server umount lustre-MDT0000 complete [ 5789.154069] LustreError: 193397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5789.187326] LustreError: 193397:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 5790.837167] LustreError: 193378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133744 with bad export cookie 14063990525752796533 [ 5790.841488] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.853576] LustreError: 193378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5791.329658] Lustre: server umount lustre-MDT0001 complete [ 5815.731255] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:36:07 (1789133767) [ 5823.068189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5823.391091] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5823.457309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5823.782386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5829.668429] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5839.646102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.755358] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5839.832352] Lustre: 199678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5845.071735] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5854.041164] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5860.254246] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5868.734363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.200281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5875.501396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5880.384347] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5890.840373] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5900.453758] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5900.625374] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5901.044823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5901.055992] Lustre: Skipped 2 previous similar messages [ 5909.320498] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 5910.690398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5917.599691] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5921.259766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.267790] Lustre: Skipped 1 previous similar message [ 5921.271914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.276273] Lustre: Skipped 3 previous similar messages [ 5926.373496] Lustre: server umount lustre-OST0000 complete [ 5926.381929] LustreError: 201758:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.395589] LustreError: 201758:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5946.861913] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5946.863972] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5946.874455] LustreError: Skipped 1 previous similar message [ 5946.901112] Lustre: Skipped 2 previous similar messages [ 5950.875872] Lustre: server umount lustre-OST0001 complete [ 5957.101018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.111545] Lustre: Skipped 4 previous similar messages [ 5969.375180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5969.571807] Lustre: server umount lustre-MDT0000 complete [ 5973.862823] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133927 with bad export cookie 14063990525752797940 [ 5973.884654] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.885749] LustreError: 199671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5974.567484] Lustre: server umount lustre-MDT0001 complete [ 5996.151672] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:39:07 (1789133947) [ 6012.073344] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 6022.561529] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 6056.302425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.547137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6056.605300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6056.660050] Lustre: lustre-MDT0000: new disk, initializing [ 6056.742893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.758990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6060.491654] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6069.459288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.590236] Lustre: 207778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6069.671643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6069.682410] Lustre: Skipped 1 previous similar message [ 6069.740271] Lustre: lustre-MDT0001: new disk, initializing [ 6069.845522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6069.858898] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6074.381259] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6082.573924] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.494855] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6093.862910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6094.192237] Lustre: lustre-OST0000: new disk, initializing [ 6094.196992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6094.207402] Lustre: 209744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6099.999306] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6101.564218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6101.570359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6101.703741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6108.106934] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6112.278927] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6112.491923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6116.648256] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6116.838088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6122.464537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.470623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6122.486199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6122.506198] Lustre: Skipped 3 previous similar messages [ 6122.522858] Lustre: Skipped 5 previous similar messages [ 6124.729972] Lustre: server umount lustre-OST0000 complete [ 6131.168940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6134.625476] Lustre: server umount lustre-MDT0000 complete [ 6137.747257] LustreError: 207772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134091 with bad export cookie 14063990525752799725 [ 6137.747896] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.755105] LustreError: 207772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6137.995651] Lustre: server umount lustre-MDT0001 complete [ 6148.013786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6148.718553] LustreError: 211959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6148.750074] LustreError: 211959:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 13 previous similar messages [ 6153.916768] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6162.172040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6166.684680] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6173.123829] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6178.625592] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6184.362476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6191.587516] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6200.873189] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6205.014578] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6205.252807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6209.428636] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6209.657709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6213.600137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6213.606617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6213.633065] Lustre: Skipped 2 previous similar messages [ 6217.528917] Lustre: server umount lustre-OST0000 complete [ 6222.777841] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6228.012487] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6257.369243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.515469] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6265.829683] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6265.847753] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6270.395881] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6275.673115] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6279.791921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.798646] Lustre: Skipped 6 previous similar messages [ 6280.878599] Lustre: server umount lustre-OST0000 complete [ 6285.281632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6285.288721] LustreError: Skipped 1 previous similar message [ 6290.774768] Lustre: server umount lustre-MDT0000 complete [ 6294.977842] LustreError: 216850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134248 with bad export cookie 14063990525752800607 [ 6294.983655] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.997070] LustreError: 216850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6295.373508] Lustre: server umount lustre-MDT0001 complete [ 6305.937520] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:44:17 (1789134257) [ 6313.024598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6314.029339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.033700] Lustre: Skipped 6 previous similar messages [ 6319.930070] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6329.599960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.917977] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6342.738765] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6348.314312] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6356.522934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.802894] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6371.790195] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6377.572618] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6377.837565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6383.596434] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6383.804864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6392.071747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.234214] Lustre: lustre-OST0001: new disk, initializing [ 6392.237798] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6392.242707] Lustre: 221296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6399.137322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6399.154469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6399.240825] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6399.290937] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6408.312513] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6412.667982] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6412.829424] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6416.589574] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6416.763657] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6419.944591] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6419.945989] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.955725] LustreError: Skipped 1 previous similar message [ 6419.976315] Lustre: Skipped 4 previous similar messages [ 6424.697882] Lustre: server umount lustre-OST0001 complete [ 6425.060406] LustreError: 219744:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.075570] LustreError: 219744:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 6430.881746] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6431.050308] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6434.946320] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6435.181683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6464.092687] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6469.638462] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6469.661793] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6469.665345] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6469.706297] Lustre: Skipped 1 previous similar message [ 6470.574127] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6478.608490] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6479.977170] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6487.732904] Lustre: server umount lustre-OST0001 complete [ 6511.924327] Lustre: server umount lustre-OST0000 complete [ 6522.204782] Lustre: server umount lustre-MDT0000 complete [ 6525.666155] LustreError: 217905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134479 with bad export cookie 14063990525752801559 [ 6525.670717] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6525.676882] LustreError: 217905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6532.015531] Lustre: server umount lustre-MDT0001 complete [ 6557.921375] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:48:29 (1789134509) [ 6565.290145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.698586] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6565.773965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6571.463047] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6581.609059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6581.694336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6581.741461] Lustre: 226542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6586.669513] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6595.426674] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6602.613748] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6612.648647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6612.925087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6621.788769] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6629.804172] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6641.164612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6641.249538] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6647.288633] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6654.316596] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6655.734768] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6660.249322] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6660.437596] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6664.688826] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6664.925936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6670.602447] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff943831547000.ost_server_uuid 50 [ 6672.338548] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff943831547000.ost_server_uuid in IDLE state after 0 sec [ 6673.381133] Lustre: Permanently deactivating lustre-OST0001 [ 6673.407373] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6677.311367] Lustre: Permanently reactivating lustre-OST0001 [ 6682.595574] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.610567] LustreError: Skipped 5 previous similar messages [ 6682.616392] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.644358] Lustre: Skipped 7 previous similar messages [ 6682.656671] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6682.663376] Lustre: Skipped 18 previous similar messages [ 6687.397646] Lustre: server umount lustre-OST0001 complete [ 6706.143394] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6706.257396] Lustre: server umount lustre-OST0000 complete [ 6716.456300] Lustre: server umount lustre-MDT0000 complete [ 6720.215652] LustreError: 226534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134673 with bad export cookie 14063990525752802854 [ 6720.224820] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6720.241158] LustreError: 226534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6720.724446] Lustre: server umount lustre-MDT0001 complete [ 6747.496847] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:51:38 (1789134698) [ 6757.306758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.641317] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6757.704871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6757.711622] Lustre: Skipped 5 previous similar messages [ 6761.932755] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6770.529202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.621557] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6770.641852] Lustre: 233235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6776.896336] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6785.456531] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6792.651533] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6800.554111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.982085] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6800.998749] Lustre: Permanently deactivating lustre-OST0000 [ 6801.021481] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6806.642954] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6806.718782] LustreError: 235368:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 6806.736986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6809.458438] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6818.034124] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6825.711932] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.838537] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6826.172340] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.180057] Lustre: Skipped 11 previous similar messages [ 6834.133125] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6834.433102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6842.697827] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6845.294124] Lustre: Permanently reactivating lustre-OST0000 [ 6845.307487] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6845.311513] Lustre: Skipped 2 previous similar messages [ 6852.718909] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6852.726890] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6852.737802] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6852.803711] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6852.829864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6867.640724] Lustre: server umount lustre-OST0001 complete [ 6886.367597] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6886.608499] Lustre: server umount lustre-OST0000 complete [ 6896.836061] Lustre: server umount lustre-MDT0000 complete [ 6900.329356] LustreError: 233229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134854 with bad export cookie 14063990525752804499 [ 6900.334374] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6900.337813] LustreError: 233229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6900.677632] Lustre: server umount lustre-MDT0001 complete [ 6911.736031] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 6914.512644] Key type lgssc unregistered [ 6914.890738] LNet: 238125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6914.898637] LNetError: 238125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6914.919930] LNet: Removed LNI 192.168.206.102@tcp [ 6916.226131] Key type .llcrypt unregistered [ 6916.228843] Key type ._llcrypt unregistered [ 6947.299294] Key type ._llcrypt registered [ 6947.301440] Key type .llcrypt registered [ 6947.399574] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 6948.266757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6948.305094] alg: No test for adler32 (adler32-zlib) [ 6949.371186] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 6949.622863] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 6951.271141] Key type lgssc registered [ 6952.366657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6962.584259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6962.629267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.559901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.865895] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6974.947273] LustreError: 239594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6974.966768] LustreError: 239594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6980.078726] LustreError: 239595:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6984.433763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.664942] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6984.703182] Lustre: 239582:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6984.714058] Lustre: Permanently deactivating lustre-MDT0001 [ 6984.724411] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6985.505922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6992.541971] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 6993.974305] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7003.474735] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7008.919772] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7016.012200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.674549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7023.428311] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7029.988165] LustreError: 241435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7030.001629] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7031.854306] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.118810] LustreError: 242056:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7035.144293] LustreError: 242056:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7035.157109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7040.028511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.219816] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.646660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7045.653673] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7046.766939] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7055.618723] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7058.992871] Lustre: Permanently reactivating lustre-MDT0001 [ 7059.040106] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7059.046347] Lustre: Skipped 2 previous similar messages [ 7064.860195] Lustre: lustre-MDT0001: Client 602d47e8-99ef-49e7-8148-e07cc522a191 (at 192.168.206.2@tcp) reconnecting [ 7066.241918] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7066.261611] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7066.269961] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7066.280097] LustreError: 243190:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7066.284920] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7066.288050] LustreError: 243190:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7066.288153] LustreError: 243190:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.102@tcp: cfg command failed: rc = -108 [ 7066.288170] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7066.288170] [ 7066.288409] LustreError: 239586:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7070.352946] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7071.821564] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7076.768427] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7077.013505] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7081.419285] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7081.664738] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7083.800573] Lustre: Permanently deactivating lustre-MDT0001 [ 7083.812497] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7083.816605] Lustre: Skipped 2 previous similar messages [ 7090.815146] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7102.946311] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7102.950173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7102.970413] Lustre: Skipped 3 previous similar messages [ 7102.972906] Lustre: Skipped 2 previous similar messages [ 7107.037735] Lustre: server umount lustre-MDT0000 complete [ 7108.069169] LustreError: 239594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7110.686715] LustreError: 243144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135064 with bad export cookie 1517268743885153017 [ 7110.698511] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7110.702037] LustreError: 243144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7111.063930] Lustre: server umount lustre-MDT0001 complete [ 7117.293576] Lustre: server umount lustre-OST0000 complete [ 7129.112449] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:58:00 (1789135080) [ 7140.331461] Lustre: server umount lustre-OST0001 complete [ 7147.465951] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 7156.961837] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7191.046292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.248866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7191.260647] Lustre: Skipped 2 previous similar messages [ 7191.305706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7191.377189] Lustre: lustre-MDT0000: new disk, initializing [ 7191.464029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.487968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7195.874318] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7204.997779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.068796] Lustre: 247876:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7205.115428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7205.119201] Lustre: Skipped 1 previous similar message [ 7205.166393] Lustre: lustre-MDT0001: new disk, initializing [ 7205.233281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7205.269983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7205.283675] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7209.808525] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7220.160485] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7228.748805] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7238.319834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.546564] Lustre: lustre-OST0000: new disk, initializing [ 7238.551953] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7238.562484] Lustre: 249843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7238.668251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.092808] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7247.946187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7247.962980] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7248.071880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7254.875603] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7256.863782] LustreError: 247883:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7264.927302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.126774] Lustre: lustre-OST0001: new disk, initializing [ 7265.137476] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7265.144534] Lustre: 250957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7266.965361] LustreError: 247883:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7266.991226] LustreError: 249436:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7271.144708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7271.179949] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7271.425234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7274.018637] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7277.047209] LustreError: 249436:0:(mdt_handler.c:3282:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7286.933725] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7289.827201] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7295.158262] Lustre: server umount lustre-OST0001 complete [ 7296.999331] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7297.001075] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7297.016904] LustreError: Skipped 1 previous similar message [ 7297.018595] LustreError: 249828:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.028287] Lustre: Skipped 1 previous similar message [ 7297.041954] LustreError: 249828:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7314.914342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7314.915808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7314.921812] LustreError: Skipped 1 previous similar message [ 7314.941265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7314.949860] Lustre: Skipped 1 previous similar message [ 7317.480606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7317.485854] LustreError: 249828:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7317.489898] Lustre: Skipped 1 previous similar message [ 7317.522982] LustreError: 249828:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 7322.598582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7322.623249] Lustre: Skipped 1 previous similar message [ 7327.200759] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7327.381396] Lustre: server umount lustre-OST0000 complete [ 7332.850225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.855900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.878931] Lustre: Skipped 2 previous similar messages [ 7332.899607] Lustre: Skipped 1 previous similar message [ 7337.615699] Lustre: server umount lustre-MDT0000 complete [ 7341.605756] LustreError: 247869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135295 with bad export cookie 1517268743885155831 [ 7341.609595] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7341.622270] LustreError: 247869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7348.232374] Lustre: server umount lustre-MDT0001 complete [ 7359.057580] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7362.314926] Key type lgssc unregistered [ 7362.680583] LNet: 253007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.698485] LNetError: 253007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.726515] LNet: Removed LNI 192.168.206.102@tcp [ 7363.508555] Key type .llcrypt unregistered [ 7363.510819] Key type ._llcrypt unregistered [ 7416.143674] Key type ._llcrypt registered [ 7416.145728] Key type .llcrypt registered [ 7416.299498] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7417.997455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7418.121040] alg: No test for adler32 (adler32-zlib) [ 7419.304550] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 7419.606038] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 7421.377400] Key type lgssc registered [ 7422.549131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7432.900614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7432.923628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7434.361445] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7434.398359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7434.612392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7440.141362] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7449.749373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.842506] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7449.866777] Lustre: 255227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7450.093507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.222754] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7464.677883] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7470.646606] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7479.439274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7479.794107] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7480.136351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.481761] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7487.648862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7495.341841] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7554.528675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7554.544135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7554.569203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.137609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7559.152347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.169938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7560.618413] Lustre: server umount lustre-OST0000 complete [ 7564.755300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7578.151681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7578.624919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7579.631602] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7579.655939] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7584.771341] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7584.788685] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7587.267560] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7595.684657] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7635.936449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7635.942768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7635.947864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7635.966945] Lustre: Skipped 1 previous similar message [ 7635.982778] Lustre: Skipped 2 previous similar messages [ 7641.058291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7641.071208] Lustre: Skipped 1 previous similar message [ 7642.228703] Lustre: server umount lustre-OST0000 complete [ 7649.760340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7649.772951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.796391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7654.995482] Lustre: server umount lustre-MDT0000 complete [ 7660.598604] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135614 with bad export cookie 3243674724384398009 [ 7660.608684] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7660.618780] LustreError: 255218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7661.364128] Lustre: server umount lustre-MDT0001 complete [ 7673.361476] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7676.208561] Key type lgssc unregistered [ 7676.508855] LNet: 260661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7676.518279] LNetError: 260661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7676.550878] LNet: Removed LNI 192.168.206.102@tcp [ 7677.563785] Key type .llcrypt unregistered [ 7677.566725] Key type ._llcrypt unregistered [ 7706.489283] Key type ._llcrypt registered [ 7706.491893] Key type .llcrypt registered [ 7706.656867] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7708.347859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7708.543009] alg: No test for adler32 (adler32-zlib) [ 7709.722640] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 7709.876288] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 7711.591257] Key type lgssc registered [ 7713.086163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7725.528379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7725.552415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7727.414898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7732.720657] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7737.840744] LustreError: 262081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7737.859972] LustreError: 262081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7742.057802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.479197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7748.280199] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7758.243700] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7766.113889] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7774.316386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.763575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.725722] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7787.625859] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7797.303196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7827.936284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7827.946158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.959405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7831.067411] Lustre: server umount lustre-OST0000 complete [ 7834.592485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7834.611062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.635130] Lustre: Skipped 1 previous similar message [ 7834.644443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7834.653291] Lustre: Skipped 1 previous similar message [ 7838.177353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.177524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.660314] Lustre: server umount lustre-MDT0000 complete [ 7843.297341] LustreError: 262081:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.326834] LustreError: 262081:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7844.432409] LustreError: 262061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135798 with bad export cookie 3917975502241748690 [ 7844.448496] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.452029] LustreError: 262061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7844.717985] Lustre: server umount lustre-MDT0001 complete [ 7855.581147] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7858.380421] Key type lgssc unregistered [ 7858.608336] LNet: 266415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7858.617230] LNetError: 266415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7858.630424] LNet: Removed LNI 192.168.206.102@tcp [ 7859.333632] Key type .llcrypt unregistered [ 7859.335625] Key type ._llcrypt unregistered [ 7876.105560] Key type ._llcrypt registered [ 7876.110317] Key type .llcrypt registered [ 7876.250320] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 7877.095690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7877.236292] alg: No test for adler32 (adler32-zlib) [ 7878.273182] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 7878.455505] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 7880.111522] Key type lgssc registered [ 7881.361445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7889.036391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7889.052575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.500829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7894.414335] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7900.646695] LustreError: 267639:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7900.680725] LustreError: 267639:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7901.643085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.045732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7905.978380] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7913.140277] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7917.875759] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7923.702086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7924.052927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.795271] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 7935.354352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7936.973855] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7947.744282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7947.749146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.759615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7949.108626] Lustre: server umount lustre-OST0000 complete [ 7953.384597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7953.395760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.418303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7955.936163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.944232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7955.993557] Lustre: Skipped 1 previous similar message [ 7961.056794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.066905] Lustre: Skipped 1 previous similar message [ 7963.222143] Lustre: server umount lustre-MDT0000 complete [ 7966.219622] LustreError: 267633:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.258730] LustreError: 267633:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7967.788201] LustreError: 267620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135921 with bad export cookie 9343619227647556051 [ 7967.798362] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7967.807040] LustreError: 267620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7968.203146] Lustre: server umount lustre-MDT0001 complete [ 7978.868439] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 7981.882604] Key type lgssc unregistered [ 7982.151817] LNet: 271316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7982.164084] LNetError: 271316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7982.183929] LNet: Removed LNI 192.168.206.102@tcp [ 7982.993194] Key type .llcrypt unregistered [ 7982.996742] Key type ._llcrypt unregistered [ 8008.051365] Key type ._llcrypt registered [ 8008.053167] Key type .llcrypt registered [ 8008.192338] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8009.430101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8009.502481] alg: No test for adler32 (adler32-zlib) [ 8010.538084] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 8010.745847] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 8012.463169] Key type lgssc registered [ 8013.678435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8021.414820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8021.454746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.164741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8028.927985] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8032.805398] LustreError: 272734:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8032.836241] LustreError: 272734:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8037.861481] LustreError: 272733:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8038.665644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8039.112317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8043.403494] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8050.661277] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8056.476746] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8062.943411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8063.423856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.509068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8068.965351] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8075.867712] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8100.322381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.336797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.349597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.395248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.398761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8103.409487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.566491] Lustre: server umount lustre-OST0000 complete [ 8111.073640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8111.078908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.088979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8111.095894] Lustre: Skipped 1 previous similar message [ 8113.639133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8113.643592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8118.757535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8118.761324] Lustre: Skipped 1 previous similar message [ 8124.384716] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8124.568647] Lustre: server umount lustre-MDT0000 complete [ 8128.526572] LustreError: 273471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136082 with bad export cookie 12854019853288510135 [ 8128.532342] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8128.548527] LustreError: 273471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8129.060731] Lustre: server umount lustre-MDT0001 complete [ 8140.197904] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 8143.197883] Key type lgssc unregistered [ 8143.503879] LNet: 277044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8143.518195] LNetError: 277044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8143.554173] LNet: Removed LNI 192.168.206.102@tcp [ 8144.628487] Key type .llcrypt unregistered [ 8144.630400] Key type ._llcrypt unregistered [ 8160.988417] Key type ._llcrypt registered [ 8160.993991] Key type .llcrypt registered [ 8161.133279] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8162.001294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8162.145251] alg: No test for adler32 (adler32-zlib) [ 8163.301749] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 8163.570092] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 8165.280365] Key type lgssc registered [ 8166.463332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8174.767766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8174.804591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8176.355511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.570139] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8186.852854] LustreError: 278284:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8186.880644] LustreError: 278284:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8189.066361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.611552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8194.091255] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8201.349950] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8207.525877] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8214.478108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8214.811187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.784926] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8222.062865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8228.144794] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8244.706411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8244.710681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.711652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8244.727623] Lustre: Skipped 1 previous similar message [ 8249.577440] Lustre: server umount lustre-OST0000 complete [ 8254.949993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.957029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.971081] Lustre: Skipped 1 previous similar message [ 8254.984967] Lustre: Skipped 2 previous similar messages [ 8260.066727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.075240] Lustre: Skipped 1 previous similar message [ 8265.185247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8265.192660] Lustre: Skipped 1 previous similar message [ 8267.231133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.592956] Lustre: server umount lustre-MDT0000 complete [ 8270.307896] LustreError: 278269:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.344583] LustreError: 278269:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8271.323893] LustreError: 279044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136225 with bad export cookie 13789980356524459231 [ 8271.330704] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8271.344264] LustreError: 279044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8271.748353] Lustre: server umount lustre-MDT0001 complete [ 8283.572510] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 8286.805743] Key type lgssc unregistered [ 8287.237506] LNet: 281989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8287.246587] LNetError: 281989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8287.282771] LNet: Removed LNI 192.168.206.102@tcp [ 8288.610654] Key type .llcrypt unregistered [ 8288.613503] Key type ._llcrypt unregistered [ 8301.158685] Key type ._llcrypt registered [ 8301.164411] Key type .llcrypt registered [ 8301.243931] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8318.683406] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8331.810516] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8332.650643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8332.690887] alg: No test for adler32 (adler32-zlib) [ 8333.698631] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 8333.981825] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 8335.695143] Key type lgssc registered [ 8336.677223] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8368.887270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8368.916871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.238613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8370.282060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8370.366545] Lustre: lustre-MDT0000: new disk, initializing [ 8370.451716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.476346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8375.049374] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8385.136490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.310657] Lustre: 285541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8385.388521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8385.393031] Lustre: Skipped 1 previous similar message [ 8385.462513] Lustre: lustre-MDT0001: new disk, initializing [ 8385.533630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8385.573322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8385.582516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8389.957657] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8399.280066] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8404.636749] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8411.068615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8411.350983] Lustre: lustre-OST0000: new disk, initializing [ 8411.353920] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8411.358191] Lustre: 287514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8411.425841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8416.808021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8416.815103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8416.968664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8417.151426] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8426.062344] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8430.101651] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8430.306984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8434.811401] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8435.024869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8437.216877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.225752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.242839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.730648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.743373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.765772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.854071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.978943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8447.984720] Lustre: Skipped 2 previous similar messages [ 8451.039156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8451.223952] Lustre: server umount lustre-OST0000 complete [ 8457.185211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8457.192385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.210811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8461.057505] Lustre: server umount lustre-MDT0000 complete [ 8465.359462] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136419 with bad export cookie 9112015523817257811 [ 8465.360526] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8465.371591] LustreError: 285535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8465.713198] Lustre: server umount lustre-MDT0001 complete [ 8476.080647] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:20:26 (1789136426) [ 8486.237333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.824304] LustreError: 289927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.849243] LustreError: 289927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8486.938284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8492.004652] LustreError: 289928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.207584] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8497.126244] LustreError: 289927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8500.777457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8501.114947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8506.271095] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8512.374257] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8516.706944] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8522.342283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.624865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.964847] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8537.833460] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8547.299531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.303133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8547.341274] Lustre: Skipped 2 previous similar messages [ 8547.364820] Lustre: Skipped 3 previous similar messages [ 8553.753457] Lustre: server umount lustre-OST0000 complete [ 8562.658130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8562.660703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8562.694234] Lustre: Skipped 1 previous similar message [ 8567.777368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8567.785516] Lustre: Skipped 5 previous similar messages [ 8572.383137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8572.600792] Lustre: server umount lustre-MDT0000 complete [ 8572.899808] LustreError: 289928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8572.918839] LustreError: 289928:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8576.143792] LustreError: 289907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136529 with bad export cookie 9112015523817258693 [ 8576.145275] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.159826] LustreError: 289907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8576.558695] Lustre: server umount lustre-MDT0001 complete [ 8587.819851] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 8590.454049] Key type lgssc unregistered [ 8590.701650] LNet: 293469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8590.713457] LNetError: 293469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.739176] LNet: Removed LNI 192.168.206.102@tcp [ 8591.594149] Key type .llcrypt unregistered [ 8591.596508] Key type ._llcrypt unregistered [ 8613.951425] Key type ._llcrypt registered [ 8613.955613] Key type .llcrypt registered [ 8614.036123] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8627.457913] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8628.457625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8628.499810] alg: No test for adler32 (adler32-zlib) [ 8629.498588] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 8629.683986] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 8631.311522] Key type lgssc registered [ 8632.324134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8668.311785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8668.326516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8669.514137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8669.557707] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8669.631318] Lustre: lustre-MDT0000: new disk, initializing [ 8669.708941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.724865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8672.481652] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8679.755477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.828968] Lustre: 297286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8679.895234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8679.898841] Lustre: Skipped 1 previous similar message [ 8679.947240] Lustre: lustre-MDT0001: new disk, initializing [ 8680.006749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8680.030000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8680.037785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8683.363182] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8690.847115] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8694.837132] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8699.757077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8699.999970] Lustre: lustre-OST2710: new disk, initializing [ 8700.005381] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8700.011477] Lustre: 299252:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8700.101852] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8704.896694] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8705.613879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8705.625466] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8705.784220] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8712.404985] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8717.538814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.667713] Lustre: lustre-OST03e8: new disk, initializing [ 8717.670823] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8717.675841] Lustre: 300316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8717.750696] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8722.638519] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8723.531781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8723.544869] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8723.622946] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8729.030292] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8734.840647] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8735.040232] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8739.480723] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8739.665879] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8744.099341] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8744.317647] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8748.419389] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8748.642236] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8754.664116] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.669062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8754.673221] Lustre: Skipped 2 previous similar messages [ 8754.681080] Lustre: Skipped 2 previous similar messages [ 8758.753767] Lustre: server umount lustre-MDT0000 complete [ 8761.907701] LustreError: 299251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136715 with bad export cookie 9042471872685807599 [ 8761.917513] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8761.918799] LustreError: 299251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8762.234177] Lustre: server umount lustre-MDT0001 complete [ 8775.988840] Lustre: server umount lustre-OST2710 complete [ 8789.388764] Lustre: server umount lustre-OST03e8 complete [ 8805.366863] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 8813.137526] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8842.173596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8842.385268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8842.409880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8842.476181] Lustre: lustre-MDT0000: new disk, initializing [ 8842.531760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8842.543814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8846.630722] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8856.750647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.872948] Lustre: 305566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8856.930565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8856.936197] Lustre: Skipped 1 previous similar message [ 8857.036096] Lustre: lustre-MDT0001: new disk, initializing [ 8857.150284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8857.192841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8857.209683] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8861.973278] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8870.392491] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8875.234423] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8881.200109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8881.550810] Lustre: lustre-OST0000: new disk, initializing [ 8881.558091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8881.567593] Lustre: 307535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8881.645840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8887.058139] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8891.439879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8891.464407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8891.600430] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8894.682765] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8898.501980] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8898.673052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8902.420699] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8902.607455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8907.235638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.237810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8907.241899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8907.264697] Lustre: Skipped 2 previous similar messages [ 8910.423936] Lustre: server umount lustre-OST0000 complete [ 8917.474380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8917.477251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8917.498127] Lustre: Skipped 1 previous similar message [ 8917.507687] Lustre: Skipped 1 previous similar message [ 8922.591953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8922.598543] Lustre: Skipped 2 previous similar messages [ 8927.715820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8927.720180] Lustre: Skipped 1 previous similar message [ 8928.223131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8928.327869] Lustre: server umount lustre-MDT0000 complete [ 8930.988756] LustreError: 305560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136884 with bad export cookie 9042471872685809279 [ 8930.997780] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8931.009036] LustreError: 305560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8931.455070] Lustre: server umount lustre-MDT0001 complete [ 8939.297146] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:28:11 (1789136891) [ 8940.586620] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8942.087621] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:28:13 (1789136893) [ 8945.517581] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 8970.887427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8971.172122] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8971.187749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8971.433821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8975.823719] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 8983.160988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.273095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8983.320456] Lustre: 311390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8987.056772] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9007.210554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.467778] Lustre: Denying initial registration attempt from nid 192.168.206.102@tcp, specified as failover [ 9007.479640] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9007.491842] LustreError: 312845:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9007.509625] LustreError: 312845:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9007.513572] LustreError: 312845:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9007.564848] Lustre: server umount lustre-OST0000 complete [ 9007.566915] LustreError: 312845:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9014.241519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9014.246544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.254561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9019.359228] Lustre: server umount lustre-MDT0000 complete [ 9019.877239] LustreError: 311403:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9019.895961] LustreError: 311403:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 9022.894972] LustreError: 311385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136976 with bad export cookie 9042471872685810161 [ 9022.899015] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9022.908457] LustreError: 311385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9023.277869] Lustre: server umount lustre-MDT0001 complete [ 9032.285419] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 9034.582238] Key type lgssc unregistered [ 9034.821464] LNet: 313971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9034.828569] LNetError: 313971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9034.841967] LNet: Removed LNI 192.168.206.102@tcp [ 9035.498394] Key type .llcrypt unregistered [ 9035.503875] Key type ._llcrypt unregistered [ 9043.189753] Key type ._llcrypt registered [ 9043.192341] Key type .llcrypt registered [ 9043.262531] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9043.921356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9043.976927] alg: No test for adler32 (adler32-zlib) [ 9045.078140] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 9045.279440] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 9046.911122] Key type lgssc registered [ 9047.852506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9089.202587] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9097.466835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9097.490255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.834131] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9098.858415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9098.996808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9103.343172] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9111.400672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9111.477975] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9111.508683] Lustre: 316733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9111.764716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9116.108717] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9122.994784] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9128.563630] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9148.009710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.202956] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9148.514410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.436833] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9161.307650] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9163.746871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9163.754706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9163.769323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.262026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9164.275131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9164.299574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.307031] Lustre: Skipped 1 previous similar message [ 9169.309588] Lustre: server umount lustre-OST0000 complete [ 9173.473311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9173.491672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9173.507475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9178.716044] Lustre: server umount lustre-MDT0000 complete [ 9182.195597] LustreError: 316726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137135 with bad export cookie 2582170727741775536 [ 9182.197945] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.207674] LustreError: 316726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9182.740941] Lustre: server umount lustre-MDT0001 complete [ 9192.529756] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 9194.947291] Key type lgssc unregistered [ 9195.250238] LNet: 320311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9195.257312] LNetError: 320311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9195.280924] LNet: Removed LNI 192.168.206.102@tcp [ 9196.016976] Key type .llcrypt unregistered [ 9196.018997] Key type ._llcrypt unregistered [ 9218.850174] Key type ._llcrypt registered [ 9218.852329] Key type .llcrypt registered [ 9218.930694] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9219.758814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9219.898267] alg: No test for adler32 (adler32-zlib) [ 9220.909424] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 9221.163300] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 9222.823626] Key type lgssc registered [ 9223.911053] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9231.245677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9231.267990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.668826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9237.110313] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9243.105834] LustreError: 321729:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.124743] LustreError: 321729:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9245.140917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.365721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9249.015032] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9255.320935] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9259.702466] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9267.060200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9267.455844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9272.748816] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9279.018072] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9285.872351] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 9286.624601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9286.636277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9286.657226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.159758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9288.160028] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9288.167452] Lustre: Skipped 1 previous similar message [ 9288.187800] Lustre: Skipped 1 previous similar message [ 9290.969883] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.2@tcp (stopping) [ 9290.975822] Lustre: Skipped 1 previous similar message [ 9292.122995] Lustre: server umount lustre-MDT0000 complete [ 9293.283878] LustreError: 321730:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9293.297240] LustreError: 321730:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9295.961878] LustreError: 321709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137249 with bad export cookie 13462277287750612515 [ 9295.972784] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.975208] LustreError: 321709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9296.086650] LustreError: 321726:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.111815] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.2@tcp (stopping) [ 9298.400195] LustreError: 321725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.404300] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.356413] Lustre: server umount lustre-MDT0001 complete [ 9307.168520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9313.826995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9321.174503] LustreError: 323560:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9321.193212] LustreError: 323560:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9321.271842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9321.322581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9324.625978] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9330.467669] LustreError: 323561:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9330.476656] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9330.486732] LustreError: 323561:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 9332.293282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9332.729260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9337.287503] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9337.831873] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9344.545509] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9345.913704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9352.573590] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9354.096363] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9362.400501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9362.405480] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9362.422307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9362.428211] Lustre: Skipped 3 previous similar messages [ 9363.440031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9374.180269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9374.343945] Lustre: server umount lustre-OST0000 complete [ 9378.784178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9378.785627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9378.794250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9378.794256] Lustre: Skipped 6 previous similar messages [ 9378.823406] Lustre: Skipped 2 previous similar messages [ 9384.219903] Lustre: server umount lustre-MDT0000 complete [ 9387.708170] LustreError: 321709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137341 with bad export cookie 13462277287750620222 [ 9387.708934] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9387.718363] LustreError: 321709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9388.103216] Lustre: server umount lustre-MDT0001 complete [ 9398.563941] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 9401.188665] Key type lgssc unregistered [ 9401.473605] LNet: 328000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9401.482973] LNetError: 328000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9401.501259] LNet: Removed LNI 192.168.206.102@tcp [ 9402.287949] Key type .llcrypt unregistered [ 9402.292361] Key type ._llcrypt unregistered [ 9425.239651] Key type ._llcrypt registered [ 9425.240964] Key type .llcrypt registered [ 9425.335518] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9426.159108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9426.232063] alg: No test for adler32 (adler32-zlib) [ 9427.314711] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 9427.496789] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 9429.162287] Key type lgssc registered [ 9430.067271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9436.557482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9436.572611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.985224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9441.700913] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9447.402623] LustreError: 329398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.416434] LustreError: 329398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9449.003766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.324853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9453.161421] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9459.528739] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9464.089092] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9467.874972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9467.877449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9467.883524] Lustre: Skipped 1 previous similar message [ 9467.892215] Lustre: Skipped 1 previous similar message [ 9472.764481] Lustre: server umount lustre-MDT0000 complete [ 9472.993510] LustreError: 330124:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9473.012315] LustreError: 330124:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9475.040510] LustreError: 329397:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9475.063294] LustreError: 329397:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9476.012817] LustreError: 329377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137429 with bad export cookie 7223232892249203039 [ 9476.014752] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9476.031366] LustreError: 329377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9476.517660] Lustre: server umount lustre-MDT0001 complete [ 9489.676649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.904199] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9489.951747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9490.201094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.070992] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9501.972626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9502.056147] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9502.088989] Lustre: 331983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9502.296159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9506.875924] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9513.252587] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9518.737561] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9529.274450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9529.544575] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9529.558529] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9529.569992] LustreError: 333867:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9529.578375] LustreError: 333867:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9529.583332] LustreError: 333867:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9529.635797] Lustre: server umount lustre-OST0000 complete [ 9529.638877] LustreError: 333867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9540.841034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9540.983169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9541.186973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9546.855137] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9547.384943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9553.619992] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9560.012964] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.140174] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9560.212393] Lustre: lustre-OST0001: new disk, initializing [ 9560.216189] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9560.223180] Lustre: 335410:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.324089] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9566.499604] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9568.849478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9568.862491] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9568.974593] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9573.898866] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9579.488986] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9579.492404] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9579.494800] LustreError: Skipped 1 previous similar message [ 9579.496625] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9579.507848] Lustre: Skipped 1 previous similar message [ 9582.852690] Lustre: server umount lustre-OST0001 complete [ 9584.624276] LustreError: 334332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9584.653733] LustreError: 334332:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9588.216057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9588.229697] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9588.259201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9588.267548] Lustre: Skipped 1 previous similar message [ 9589.731094] LustreError: 334732:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.776636] LustreError: 334732:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9592.903528] Lustre: server umount lustre-OST0000 complete [ 9599.456264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9599.473045] LustreError: Skipped 1 previous similar message [ 9599.483316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9599.504232] Lustre: Skipped 1 previous similar message [ 9599.515937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9599.526097] Lustre: Skipped 3 previous similar messages [ 9603.906701] Lustre: server umount lustre-MDT0000 complete [ 9605.091371] LustreError: 331996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.107290] LustreError: 331996:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9607.525022] LustreError: 331975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137561 with bad export cookie 7223232892249203592 [ 9607.527405] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9607.533712] LustreError: 331975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9607.843890] Lustre: server umount lustre-MDT0001 complete [ 9618.478814] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [ 9621.239968] Key type lgssc unregistered [ 9621.579785] LNet: 337368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9621.593366] LNetError: 337368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9621.635898] LNet: Removed LNI 192.168.206.102@tcp [ 9622.596849] Key type .llcrypt unregistered [ 9622.601057] Key type ._llcrypt unregistered [ 9683.221339] Key type ._llcrypt registered [ 9683.224537] Key type .llcrypt registered [ 9683.291304] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [ 9695.669144] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [ 9696.686765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9696.706772] alg: No test for adler32 (adler32-zlib) [ 9697.743818] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [ 9697.954616] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [ 9699.623138] Key type lgssc registered [ 9700.440728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9729.172689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9729.196134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9730.431502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9730.467301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9730.553394] Lustre: lustre-MDT0000: new disk, initializing [ 9730.628187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9730.651728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9734.149935] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9743.501418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9743.640959] Lustre: 342280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9743.686442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9743.690392] Lustre: Skipped 1 previous similar message [ 9743.782304] Lustre: lustre-MDT0001: new disk, initializing [ 9743.869242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.891538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9743.903660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9748.192397] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9758.758357] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9765.784236] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9772.666035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9773.003666] Lustre: lustre-OST0000: new disk, initializing [ 9773.007366] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9773.028100] Lustre: 344250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9773.140751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9779.310912] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9779.773600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9779.782625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9779.882468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9787.577544] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9791.694636] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9791.877787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9796.808942] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9797.019930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9800.676788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9800.685152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9800.692543] LustreError: Skipped 1 previous similar message [ 9800.696489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9800.707020] Lustre: Skipped 1 previous similar message [ 9800.733398] Lustre: Skipped 1 previous similar message [ 9805.800652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9805.805774] Lustre: Skipped 1 previous similar message [ 9810.913452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9810.923579] Lustre: Skipped 1 previous similar message [ 9812.959595] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9813.152499] Lustre: server umount lustre-OST0000 complete [ 9820.640311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9820.659751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.679511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.687750] Lustre: server umount lustre-MDT0000 complete [ 9826.412969] LustreError: 342272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137780 with bad export cookie 16250856652854283893 [ 9826.426615] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.808980] Lustre: server umount lustre-MDT0001 complete [ 9834.890354] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:43:06 (1789137786) [ 9842.058527] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:43:13 (1789137793) [ 9849.866632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9850.231447] LustreError: 347006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9850.253337] LustreError: 347006:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9850.309640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.437551] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9855.462521] LustreError: 347007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.583855] LustreError: 347006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.143696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.531496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.513296] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9875.104497] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9880.532498] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9888.012265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9888.468582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9894.817286] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9901.132851] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9908.973594] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9910.784433] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9912.325509] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9913.933920] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9915.458973] Lustre: Failing over lustre-MDT0000 [ 9915.672932] Lustre: server umount lustre-MDT0000 complete [ 9916.387093] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9916.389191] LustreError: 347001:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9916.412789] Lustre: Skipped 3 previous similar messages [ 9916.444734] LustreError: 347001:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9921.507135] LustreError: 348303:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9921.527943] LustreError: 348303:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9931.745330] LustreError: 347007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9931.759380] LustreError: 347007:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 9932.774042] Lustre: 340504:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137870/real 1789137870] req@0000000099acbfc8 x1876046801605504/t0(0) o400->MGC192.168.206.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789137886 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9932.799642] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9933.214643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.031432] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe186a406692c561d [ 9943.063995] Lustre: MGC192.168.206.102@tcp: Connection restored to 0@lo (at 0@lo) [ 9943.425397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.501382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9944.798165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9948.711369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9948.820524] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9948.941093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9950.252241] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [ 9961.323951] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9963.717710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9967.137879] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9968.908270] Lustre: Failing over lustre-MDT0000 [ 9969.136209] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9969.144201] LustreError: 348303:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9969.160977] Lustre: Skipped 1 previous similar message [ 9969.188819] LustreError: 348303:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 9969.321549] Lustre: server umount lustre-MDT0000 complete [ 9984.481742] Lustre: 340503:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789137922/real 1789137922] req@00000000d5b07c92 x1876046801627264/t0(0) o400->MGC192.168.206.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789137938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9984.525324] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9988.313136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9994.724666] LustreError: 340502:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fabe10a3 x1876046801632768/t0(0) o250->MGC192.168.206.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9994.976592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9995.026826] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9997.019897] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9998.857928] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10000.368656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10000.374589] Lustre: Skipped 2 previous similar messages [10000.400704] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10000.451700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10007.656905] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10009.078740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10015.712396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.717860] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.736860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.755267] Lustre: Skipped 1 previous similar message [10015.784925] Lustre: Skipped 4 previous similar messages [10020.117256] Lustre: server umount lustre-MDT0000 complete [10020.832414] LustreError: 347006:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10020.855642] LustreError: 347006:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [10023.009419] LustreError: 346988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137976 with bad export cookie 16250856652854286616 [10023.016454] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10023.021118] LustreError: 346988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10023.242600] Lustre: server umount lustre-MDT0001 complete [10035.418200] Lustre: server umount lustre-OST0000 complete [10048.740629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.047474] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10060.365692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.654421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10060.658805] Lustre: Skipped 1 previous similar message [10064.050765] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10068.667871] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10073.016539] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10078.497375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.203154] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10088.735761] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10094.138518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10099.170256] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10099.179347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10099.192632] Lustre: Skipped 1 previous similar message [10099.208090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10099.213762] Lustre: Skipped 1 previous similar message [10103.828583] Lustre: server umount lustre-OST0000 complete [10113.316631] Lustre: server umount lustre-MDT0000 complete [10114.528298] LustreError: 354072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.538673] LustreError: 354072:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10115.988906] LustreError: 353326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138069 with bad export cookie 16250856652854287365 [10115.993310] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10115.997256] LustreError: 353326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10116.313215] Lustre: server umount lustre-MDT0001 complete [10125.401591] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [10127.774624] Key type lgssc unregistered [10128.001866] LNet: 356877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10128.012586] LNetError: 356877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10128.029777] LNet: Removed LNI 192.168.206.102@tcp [10128.725977] Key type .llcrypt unregistered [10128.729862] Key type ._llcrypt unregistered [10148.422167] Key type ._llcrypt registered [10148.425041] Key type .llcrypt registered [10148.522611] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10149.453330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10149.476942] alg: No test for adler32 (adler32-zlib) [10150.502894] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [10150.746764] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [10152.432049] Key type lgssc registered [10153.387983] Lustre: Echo OBD driver; http://www.lustre.org/ [10159.540403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10159.553287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.849652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10164.261691] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10170.340454] LustreError: 358375:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10170.368454] LustreError: 358375:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10170.719861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10171.048613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10174.676574] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10180.019257] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10184.270736] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10189.121837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10189.370687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10193.319464] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10198.140638] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10205.747844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10217.440628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10217.452275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10217.462389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10219.723238] Lustre: server umount lustre-MDT0000 complete [10221.034211] LustreError: 359470:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10221.035474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.052740] LustreError: 359470:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10221.065348] Lustre: Skipped 1 previous similar message [10222.782719] LustreError: 358353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138176 with bad export cookie 17719121249888304555 [10222.785722] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10222.802575] LustreError: 358353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10223.130684] Lustre: server umount lustre-MDT0001 complete [10236.639348] Lustre: server umount lustre-OST0000 complete [10245.754639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10246.058616] LustreError: 362104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10246.075087] LustreError: 362104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10246.114683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10249.483121] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10251.237698] LustreError: 362105:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.329290] LustreError: 362104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10255.528544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10255.810714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10259.251446] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10266.274281] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10271.176762] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10276.953155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.291075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10282.682383] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10288.388652] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10290.608780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10306.019955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10306.025759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10306.035574] Lustre: Skipped 1 previous similar message [10306.049976] Lustre: Skipped 2 previous similar messages [10311.127487] Lustre: server umount lustre-MDT0000 complete [10311.138562] LustreError: 363574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10311.157387] LustreError: 363574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10315.715445] LustreError: 362085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138269 with bad export cookie 17719121249888305668 [10315.720927] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10315.730400] LustreError: 362085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10316.251623] Lustre: server umount lustre-MDT0001 complete [10331.034213] Lustre: server umount lustre-OST0000 complete [10344.597367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10344.959695] LustreError: 366012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10344.985964] LustreError: 366012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10345.060738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10349.129264] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10357.079144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10361.565273] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10369.601846] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10375.244944] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10380.730690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10381.024925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10381.033419] Lustre: Skipped 1 previous similar message [10386.494417] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10389.306804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10393.580790] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10398.688388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10398.697225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.704733] Lustre: Skipped 1 previous similar message [10398.708522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10399.737524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10404.191655] Lustre: server umount lustre-OST0000 complete [10408.423771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10408.439964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.458451] Lustre: Skipped 1 previous similar message [10408.464643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.474422] Lustre: Skipped 2 previous similar messages [10415.073042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10415.077931] Lustre: Skipped 3 previous similar messages [10422.239682] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10422.505154] Lustre: server umount lustre-MDT0000 complete [10424.301649] LustreError: 366013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.334502] LustreError: 366013:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10426.622712] LustreError: 365992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138380 with bad export cookie 17719121249888306690 [10426.626384] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10426.629714] LustreError: 365992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10426.979917] Lustre: server umount lustre-MDT0001 complete [10438.889968] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [10442.050591] Key type lgssc unregistered [10442.487214] LNet: 369551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10442.504782] LNetError: 369551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10443.557481] LNet: Removed LNI 192.168.206.102@tcp [10444.751051] Key type .llcrypt unregistered [10444.752886] Key type ._llcrypt unregistered [10470.167329] Key type ._llcrypt registered [10470.169296] Key type .llcrypt registered [10470.302385] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10471.724065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10471.775352] alg: No test for adler32 (adler32-zlib) [10473.096472] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [10473.553833] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [10475.319352] Key type lgssc registered [10476.946582] Lustre: Echo OBD driver; http://www.lustre.org/ [10484.070793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10484.100305] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.111123] LustreError: 370999:0:(osd_handler.c:8517:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10484.128866] LustreError: 370999:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10484.138413] LustreError: 370999:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10484.145852] LustreError: 370999:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10484.155289] LustreError: 370999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10490.936612] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.941165] LustreError: 371396:0:(osd_handler.c:8517:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10490.957891] LustreError: 371396:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10490.967302] LustreError: 371396:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10490.973715] LustreError: 371396:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10490.985211] LustreError: 371396:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10506.155975] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [10508.452876] Key type lgssc unregistered [10508.735594] LNet: 372402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10508.743669] LNetError: 372402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10508.765453] LNet: Removed LNI 192.168.206.102@tcp [10509.488685] Key type .llcrypt unregistered [10509.495472] Key type ._llcrypt unregistered [10525.832158] Key type ._llcrypt registered [10525.833984] Key type .llcrypt registered [10525.921759] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [10538.517746] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10539.217978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10539.331428] alg: No test for adler32 (adler32-zlib) [10540.486475] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [10540.831852] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [10542.543173] Key type lgssc registered [10544.073818] Lustre: Echo OBD driver; http://www.lustre.org/ [10579.304913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10579.332382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10580.543251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10580.574431] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10580.698642] Lustre: lustre-MDT0000: new disk, initializing [10580.765410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10580.786151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10584.838689] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10594.538113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.644230] Lustre: 375537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10594.768988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10594.779992] Lustre: Skipped 1 previous similar message [10594.905171] Lustre: lustre-MDT0001: new disk, initializing [10595.022754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10595.096136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10595.112107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10599.138435] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10607.492691] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10612.722387] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10619.140062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10619.436265] Lustre: lustre-OST0000: new disk, initializing [10619.440300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10619.451560] Lustre: 377505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10619.557579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10626.813444] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10629.745097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10629.759315] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10629.888085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10635.882985] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10641.142609] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10641.323950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10646.237547] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10646.541837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10650.592963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10650.613209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10650.644661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10655.718722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10655.730700] Lustre: Skipped 1 previous similar message [10660.834861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10660.846266] Lustre: Skipped 2 previous similar messages [10662.879148] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10663.095379] Lustre: server umount lustre-OST0000 complete [10666.976396] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10666.988920] LustreError: Skipped 1 previous similar message [10666.995246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10667.003066] Lustre: Skipped 1 previous similar message [10667.007334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.073123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10671.074439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10672.829563] Lustre: server umount lustre-MDT0000 complete [10676.758419] LustreError: 377504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138630 with bad export cookie 12464810269453523966 [10676.765907] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10676.776667] LustreError: 377504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10677.011659] Lustre: server umount lustre-MDT0001 complete [10684.825939] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10686.713624] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:57:18 (1789138638) [10693.591206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.061782] LustreError: 379965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10694.082333] LustreError: 379965:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10694.152400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10698.691107] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10699.235767] LustreError: 379966:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10703.332893] LustreError: 379965:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.955915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10707.326814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10711.418182] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10718.014575] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10722.995888] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10729.672580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10730.104421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10736.243217] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10743.622662] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10750.130675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10750.286330] Lustre: lustre-OST0001: new disk, initializing [10750.289534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10750.297819] Lustre: 382800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10750.386662] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10755.940757] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10756.712486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10756.720754] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10756.806587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10764.826446] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10771.722177] Lustre: lustre-OST0001: Not available for connect from 192.168.206.2@tcp (stopping) [10771.729376] Lustre: Skipped 1 previous similar message [10772.452192] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10772.459540] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10772.474227] Lustre: Skipped 1 previous similar message [10774.387621] Lustre: server umount lustre-OST0001 complete [10776.797716] LustreError: 381787:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.701729] LustreError: 381789:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.722867] LustreError: 381789:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10792.938980] LustreError: 381789:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.941212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10792.947615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10792.947623] Lustre: Skipped 2 previous similar messages [10792.977633] LustreError: 381789:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10793.013991] Lustre: Skipped 1 previous similar message [10797.935436] Lustre: server umount lustre-OST0000 complete [10803.172385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10808.311833] Lustre: server umount lustre-MDT0000 complete [10812.074144] LustreError: 379947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138765 with bad export cookie 12464810269453524848 [10812.080134] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10812.085110] LustreError: 379947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10818.772803] Lustre: server umount lustre-MDT0001 complete [10831.015454] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [10835.773684] Key type lgssc unregistered [10836.225937] LNet: 384795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10836.234911] LNetError: 384795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10836.255695] LNet: Removed LNI 192.168.206.102@tcp [10838.053658] Key type .llcrypt unregistered [10838.058534] Key type ._llcrypt unregistered [10893.087092] Key type ._llcrypt registered [10893.089574] Key type .llcrypt registered [10893.186424] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [10894.323288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10894.443563] alg: No test for adler32 (adler32-zlib) [10895.454725] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [10895.712412] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [10897.415271] Key type lgssc registered [10899.097521] Lustre: Echo OBD driver; http://www.lustre.org/ [10908.894965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10908.916914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.267703] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10910.316580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10910.525337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10915.106235] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10924.207916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.371972] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10924.409541] Lustre: 387622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10924.645416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10929.562025] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10936.685887] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10942.185707] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10944.999731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10945.008762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10945.023505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10948.064866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.066635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10950.427322] Lustre: server umount lustre-MDT0000 complete [10954.341190] LustreError: 388373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138908 with bad export cookie 1217074691171306578 [10954.354172] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10954.356153] LustreError: 388373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10954.666035] Lustre: server umount lustre-MDT0001 complete [10963.969546] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:01:55 (1789138915) [10970.616699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10971.303272] LustreError: 390134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10971.338172] LustreError: 390134:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10971.399953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10976.279283] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10976.737593] LustreError: 390135:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.857826] LustreError: 390134:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.372832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10985.758082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10991.064801] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [10999.057444] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11005.081914] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11011.591253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11011.821562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11012.035688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11018.081409] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11025.763329] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11033.407948] Lustre: Permanently deactivating lustre-OST0000 [11033.442137] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11042.838173] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11045.666619] LustreError: 393170:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11045.679488] LustreError: 393170:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11045.689419] LustreError: 393170:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11048.975468] LustreError: 393219:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11048.978298] LustreError: 393219:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11057.383346] Lustre: server umount lustre-OST0000 complete [11059.896611] LustreError: 393468:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11059.906228] LustreError: 393468:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11062.752772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11062.768936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11062.791227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11062.801141] Lustre: Skipped 1 previous similar message [11064.805943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11064.808897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11064.843449] Lustre: Skipped 1 previous similar message [11069.921416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11069.928516] Lustre: Skipped 1 previous similar message [11072.089541] Lustre: server umount lustre-MDT0000 complete [11075.043902] LustreError: 390130:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.080390] LustreError: 390130:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11076.080115] LustreError: 390864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139029 with bad export cookie 1217074691171307159 [11076.086712] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11076.099269] LustreError: 390864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11085.497410] Lustre: server umount lustre-MDT0001 complete [11094.215848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.219702] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11101.976645] LustreError: 394627:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11108.076436] Lustre: 394967:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.102@tcp [11110.453426] Lustre: server umount MGS complete [11122.332236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.704610] LustreError: 395693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11122.731918] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11122.740228] Lustre: Skipped 1 previous similar message [11122.855297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11127.548839] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11133.410096] LustreError: 395694:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.439888] LustreError: 395694:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11136.426205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.686333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11141.980541] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11149.227462] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11154.631336] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11156.203324] Lustre: Permanently reactivating lustre-OST0000 [11156.213880] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11156.219119] Lustre: Skipped 2 previous similar messages [11185.119863] LustreError: 397520:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11191.508771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11191.777288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11191.781913] Lustre: Skipped 1 previous similar message [11197.661673] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11205.086689] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11215.339219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11215.362872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11215.363586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11215.375899] LustreError: Skipped 1 previous similar message [11215.379601] Lustre: Skipped 1 previous similar message [11217.640799] Lustre: server umount lustre-OST0000 complete [11224.032404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11224.058368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11224.085312] Lustre: Skipped 1 previous similar message [11228.890680] Lustre: server umount lustre-MDT0000 complete [11230.692881] LustreError: 395694:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.717548] LustreError: 395694:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11232.505366] LustreError: 397264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139186 with bad export cookie 1217074691171308209 [11232.506845] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11232.517369] LustreError: 397264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11232.834504] Lustre: server umount lustre-MDT0001 complete [11242.904771] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11245.402275] Key type lgssc unregistered [11245.670160] LNet: 400156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11245.686233] LNetError: 400156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11245.708195] LNet: Removed LNI 192.168.206.102@tcp [11246.700275] Key type .llcrypt unregistered [11246.702458] Key type ._llcrypt unregistered [11263.001378] Key type ._llcrypt registered [11263.002799] Key type .llcrypt registered [11263.111237] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [11280.111425] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11280.947988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11280.982805] alg: No test for adler32 (adler32-zlib) [11281.989695] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [11282.190378] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11284.015173] Key type lgssc registered [11285.235964] Lustre: Echo OBD driver; http://www.lustre.org/ [11317.557879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11317.592689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.901206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11318.961861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11319.091088] Lustre: lustre-MDT0000: new disk, initializing [11319.180389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11319.198361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11323.214576] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11332.330448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11332.436591] Lustre: 403312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11332.484247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11332.488324] Lustre: Skipped 1 previous similar message [11332.552168] Lustre: lustre-MDT0001: new disk, initializing [11332.652973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11332.694492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11332.703136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11336.557274] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11345.296786] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11350.648915] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11356.555622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11356.780439] Lustre: lustre-OST0000: new disk, initializing [11356.784338] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11356.790177] Lustre: 405281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11356.868267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.486234] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11364.405539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11364.419313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11364.492531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11371.916402] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11377.058693] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11377.244458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11381.567942] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11381.769159] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11384.810840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.814484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11384.833811] Lustre: Skipped 1 previous similar message [11384.858018] Lustre: Skipped 1 previous similar message [11389.922923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11389.944537] Lustre: Skipped 1 previous similar message [11395.042842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11395.057753] Lustre: Skipped 1 previous similar message [11398.111128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11398.318278] Lustre: server umount lustre-OST0000 complete [11404.256647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11404.268818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11404.297200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11405.283398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.033423] Lustre: server umount lustre-MDT0000 complete [11413.530068] LustreError: 405280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139367 with bad export cookie 7054494628614891096 [11413.534592] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11413.554791] LustreError: 405280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11414.045424] Lustre: server umount lustre-MDT0001 complete [11422.183773] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11423.751591] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:09:35 (1789139375) [11430.217063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.722037] LustreError: 407745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.749325] LustreError: 407745:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11430.822992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11435.670250] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11436.002141] LustreError: 407746:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11441.120291] LustreError: 407745:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.904461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.342752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11449.936223] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11457.687953] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11463.621621] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.233616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11470.551633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11477.608940] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11484.215773] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11486.045349] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11488.979619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11496.420928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.430273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11496.432357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11496.440735] Lustre: Skipped 3 previous similar messages [11500.599867] Lustre: server umount lustre-OST0000 complete [11506.663299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.665712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11506.670773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11506.677865] Lustre: Skipped 2 previous similar messages [11510.589786] Lustre: server umount lustre-MDT0000 complete [11511.776857] LustreError: 407742:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.788277] LustreError: 407742:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11513.430284] LustreError: 407726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139467 with bad export cookie 7054494628614891978 [11513.442060] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11513.445475] LustreError: 407726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11513.758711] Lustre: server umount lustre-MDT0001 complete [11522.735684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11525.155863] Key type lgssc unregistered [11525.495153] LNet: 411474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11525.501871] LNetError: 411474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11525.520750] LNet: Removed LNI 192.168.206.102@tcp [11526.361250] Key type .llcrypt unregistered [11526.363202] Key type ._llcrypt unregistered [11547.095586] Key type ._llcrypt registered [11547.097515] Key type .llcrypt registered [11547.190816] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11547.496146] Key type .llcrypt unregistered [11547.497976] Key type ._llcrypt unregistered [11563.013854] Key type ._llcrypt registered [11563.016349] Key type .llcrypt registered [11563.119513] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11564.356496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11564.406292] alg: No test for adler32 (adler32-zlib) [11565.569418] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [11565.880452] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11567.520508] Key type lgssc registered [11568.692110] Lustre: Echo OBD driver; http://www.lustre.org/ [11574.916609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11574.930773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11576.272128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11580.322237] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11586.530966] LustreError: 413693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.555678] LustreError: 413693:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11587.788747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11588.079943] LustreError: 413692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11588.108735] LustreError: 413692:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11588.130116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.197326] LustreError: 413693:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.237151] LustreError: 413693:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11593.655961] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11598.307019] LustreError: 413692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.327440] LustreError: 413692:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11600.383799] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11601.400872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11606.517205] LustreError: 415007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11606.542259] LustreError: 415007:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11606.570294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11606.963316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11611.625830] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11621.857202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11621.874659] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.901047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11622.379665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11622.395393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.416144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11622.422039] Lustre: Skipped 1 previous similar message [11627.222835] Lustre: server umount lustre-OST0000 complete [11632.610046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11632.616122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.619899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11632.641962] Lustre: Skipped 1 previous similar message [11637.734734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11637.748981] Lustre: Skipped 2 previous similar messages [11638.186789] Lustre: server umount lustre-MDT0000 complete [11642.376638] LustreError: 413673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139596 with bad export cookie 5940534845822425180 [11642.382948] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11642.399143] LustreError: 413673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11642.940403] Lustre: server umount lustre-MDT0001 complete [11654.251922] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11657.073110] Key type lgssc unregistered [11657.509246] LNet: 416842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11657.526913] LNetError: 416842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11657.551983] LNet: Removed LNI 192.168.206.102@tcp [11658.326237] Key type .llcrypt unregistered [11658.331452] Key type ._llcrypt unregistered [11679.475719] Key type ._llcrypt registered [11679.479413] Key type .llcrypt registered [11679.571759] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11680.355148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11680.425132] alg: No test for adler32 (adler32-zlib) [11681.482053] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [11681.730250] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11683.399574] Key type lgssc registered [11684.482338] Lustre: Echo OBD driver; http://www.lustre.org/ [11690.975957] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11690.990633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11706.527396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11711.649585] LustreError: 418212:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.102@tcp: failed processing log, type 4: rc = -110 [11738.399412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11744.268608] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11750.370905] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11755.655890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11756.022970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11756.083242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11759.932736] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11761.120851] LustreError: 418237:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11761.129491] LustreError: 418237:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11766.240586] LustreError: 418238:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11767.359380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11767.743503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11772.705622] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11779.419754] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11784.445915] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11793.380199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11793.386860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11793.389102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11793.451485] Lustre: Skipped 1 previous similar message [11798.507449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11798.532428] Lustre: Skipped 1 previous similar message [11803.618566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11803.624069] Lustre: Skipped 1 previous similar message [11807.199216] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11807.854069] Lustre: server umount lustre-OST0000 complete [11813.866043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11813.873962] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11813.911299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11813.924944] Lustre: Skipped 2 previous similar messages [11818.902450] Lustre: server umount lustre-MDT0000 complete [11818.979496] LustreError: 419250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.998498] LustreError: 419250:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11822.266406] LustreError: 418219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139776 with bad export cookie 5707321634614650945 [11822.270632] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11822.276124] LustreError: 418219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11822.637641] Lustre: server umount lustre-MDT0001 complete [11831.132712] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11833.674787] Key type lgssc unregistered [11834.055860] LNet: 421786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11834.066878] LNetError: 421786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11834.110785] LNet: Removed LNI 192.168.206.102@tcp [11834.781517] Key type .llcrypt unregistered [11834.784849] Key type ._llcrypt unregistered [11857.226852] Key type ._llcrypt registered [11857.232605] Key type .llcrypt registered [11857.359078] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11858.459518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11858.478193] alg: No test for adler32 (adler32-zlib) [11859.490277] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [11859.678344] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11861.327168] Key type lgssc registered [11862.209560] Lustre: Echo OBD driver; http://www.lustre.org/ [11868.667881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11868.691918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.143733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11874.141370] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11879.396412] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.415566] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11882.441152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.814782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11886.964842] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11893.303478] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11898.983507] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11905.606865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11905.827327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11912.125825] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [11915.051266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11919.304929] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11923.936310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11923.960115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.994120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.475563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11925.478139] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.130359] Lustre: server umount lustre-MDT0000 complete [11930.595055] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11930.629863] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11931.103052] LustreError: 423185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.2@tcp arrived at 1789139884 with bad export cookie 14453956041623631953 [11931.120433] LustreError: 423185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11932.919531] LustreError: 423939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789139886 with bad export cookie 14453956041623631057 [11932.925055] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11932.927649] LustreError: 423939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11933.667652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11933.675534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11933.694681] Lustre: Skipped 1 previous similar message [11933.700462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11933.706696] Lustre: Skipped 3 previous similar messages [11935.717583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11935.722322] LustreError: 424259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11935.744357] LustreError: 424259:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11937.760950] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.787202] LustreError: 423204:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11939.154628] Lustre: server umount lustre-OST0000 complete [11942.883029] LustreError: 423201:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.902630] LustreError: 423201:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11944.831209] Lustre: server umount lustre-MDT0001 complete [11954.012955] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [11956.675373] Key type lgssc unregistered [11956.937597] LNet: 426892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11956.947631] LNetError: 426892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11956.981521] LNet: Removed LNI 192.168.206.102@tcp [11957.813973] Key type .llcrypt unregistered [11957.820388] Key type ._llcrypt unregistered [11983.313500] Key type ._llcrypt registered [11983.315929] Key type .llcrypt registered [11983.494412] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [11984.701977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11984.759795] alg: No test for adler32 (adler32-zlib) [11985.983477] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [11986.245072] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [11987.935350] Key type lgssc registered [11989.511241] Lustre: Echo OBD driver; http://www.lustre.org/ [11998.039231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11998.089850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11999.493428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12004.701084] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12009.958275] LustreError: 428309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.984842] LustreError: 428309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12013.992404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.509700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12018.879564] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12025.840087] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12031.046176] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12037.902110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12038.301294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12044.422241] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12044.464717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12051.227554] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12055.522817] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12055.531546] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.551165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12058.868560] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.2@tcp (stopping) [12060.150272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12060.153782] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12061.733855] Lustre: server umount lustre-MDT0001 complete [12065.253085] LustreError: 428309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.282458] LustreError: 428309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12070.368149] LustreError: 428310:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.369800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.371773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12070.419063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12070.428313] Lustre: Skipped 1 previous similar message [12073.712235] Lustre: server umount lustre-OST0000 complete [12075.490322] LustreError: 428309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12077.149964] Lustre: server umount lustre-MDT0000 complete [12088.049860] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12090.638155] Key type lgssc unregistered [12090.988965] LNet: 431996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12091.001333] LNetError: 431996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12091.035526] LNet: Removed LNI 192.168.206.102@tcp [12092.025570] Key type .llcrypt unregistered [12092.028208] Key type ._llcrypt unregistered [12120.737784] Key type ._llcrypt registered [12120.739283] Key type .llcrypt registered [12120.836916] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [12137.373556] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [12139.168077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12139.241433] alg: No test for adler32 (adler32-zlib) [12140.410431] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [12140.770333] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [12142.439148] Key type lgssc registered [12143.529623] Lustre: Echo OBD driver; http://www.lustre.org/ [12175.399258] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [12188.781719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12188.803989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12190.186045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12190.238490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12190.320289] Lustre: lustre-MDT0000: new disk, initializing [12190.386353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12190.406734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12194.948561] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12210.347229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12210.594921] Lustre: lustre-OST0000: new disk, initializing [12210.598369] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12210.602619] Lustre: Skipped 1 previous similar message [12210.607308] Lustre: 437028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12210.666251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12216.385120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12216.393714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12216.414660] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12216.490207] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12225.523476] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12238.911037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.029669] Lustre: 436008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12239.101256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12239.189207] Lustre: lustre-MDT0001: new disk, initializing [12239.305178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12239.352071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12239.361415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12243.372147] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12251.594784] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12251.806347] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12255.674966] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12255.909216] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12260.077504] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.2@tcp (stopping) [12260.834301] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12260.835935] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.840126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12260.846386] Lustre: Skipped 1 previous similar message [12265.176081] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.2@tcp (stopping) [12265.182053] Lustre: Skipped 1 previous similar message [12266.204622] Lustre: server umount lustre-MDT0001 complete [12276.200389] LustreError: 436016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.218080] LustreError: 436016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12281.316271] LustreError: 438682:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12281.327522] LustreError: 438682:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12282.848219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12282.857424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12282.875515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12282.879484] Lustre: Skipped 2 previous similar messages [12286.436710] LustreError: 436021:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12288.775662] Lustre: server umount lustre-OST0000 complete [12291.554330] LustreError: 438724:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12306.399189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12308.657777] Lustre: server umount lustre-MDT0000 complete [12320.668986] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12322.900611] Key type lgssc unregistered [12323.114163] LNet: 440781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12323.119673] LNetError: 440781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12323.134394] LNet: Removed LNI 192.168.206.102@tcp [12323.775209] Key type .llcrypt unregistered [12323.781362] Key type ._llcrypt unregistered [12360.403861] Key type ._llcrypt registered [12360.408740] Key type .llcrypt registered [12360.508568] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [12360.772540] Key type .llcrypt unregistered [12360.774180] Key type ._llcrypt unregistered [12379.203583] Key type ._llcrypt registered [12379.209660] Key type .llcrypt registered [12379.301400] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [12380.450672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12380.478562] alg: No test for adler32 (adler32-zlib) [12381.482371] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [12381.643700] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [12383.303239] Key type lgssc registered [12384.244934] Lustre: Echo OBD driver; http://www.lustre.org/ [12409.013925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12409.058662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12410.234756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12410.273453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12410.427477] Lustre: lustre-MDT0000: new disk, initializing [12410.578364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12410.613395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12416.496898] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12428.871731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12429.077441] Lustre: 444163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12429.237952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12429.246443] Lustre: Skipped 1 previous similar message [12429.397602] Lustre: lustre-MDT0001: new disk, initializing [12429.551263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12429.605713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12429.618435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12434.814333] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12443.650754] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12448.039456] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12453.203727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12453.435830] Lustre: lustre-OST0000: new disk, initializing [12453.440583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12453.446942] Lustre: 446131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12453.522292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12458.450159] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12462.672119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12462.680162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12462.772131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12466.456355] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12473.314933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12473.319405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12473.338609] Lustre: Skipped 1 previous similar message [12478.360703] Lustre: server umount lustre-MDT0000 complete [12481.902266] LustreError: 444968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140435 with bad export cookie 6465428370044095262 [12481.913245] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12481.919454] LustreError: 444968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12482.245256] Lustre: server umount lustre-MDT0001 complete [12495.987816] Lustre: server umount lustre-OST0000 complete [12506.943924] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:27:38 (1789140458) [12514.819176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.513702] LustreError: 448158:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12515.533659] LustreError: 448158:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12515.594630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12520.799335] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12520.930841] LustreError: 448159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12526.049469] LustreError: 448158:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.172422] LustreError: 448159:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12531.226762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12531.622668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12535.481881] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12542.016363] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12546.696777] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12565.865193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12566.075484] Lustre: Found index 0 for lustre-OST0000, updating log [12566.223517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12570.968290] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [12576.727633] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12586.741489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12592.108673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12592.115437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12592.137555] Lustre: Skipped 1 previous similar message [12592.141453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12592.161390] Lustre: Skipped 3 previous similar messages [12593.185682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12598.232407] Lustre: server umount lustre-OST0000 complete [12614.624648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12614.629856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12614.640745] Lustre: Skipped 2 previous similar messages [12614.644313] Lustre: Skipped 1 previous similar message [12619.745805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12619.765103] Lustre: Skipped 2 previous similar messages [12620.771834] LustreError: 448158:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12620.786118] LustreError: 448158:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12620.860955] Lustre: server umount lustre-MDT0000 complete [12625.157904] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789140578 with bad export cookie 6465428370044096655 [12625.159933] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12625.179692] LustreError: 448139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12625.875399] Lustre: server umount lustre-MDT0001 complete [12631.583141] LNet: 1 local NIs in recovery (showing 1): 192.168.206.102@tcp [12634.924950] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:29:46 (1789140586) [13254.036312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13254.487183] LustreError: 454594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13254.512485] LustreError: 454594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13254.567384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13258.327272] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13259.761632] LustreError: 454595:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13264.878030] LustreError: 454594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13268.331900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13269.049974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13275.060775] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13283.990676] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13290.177547] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13309.471227] LNet: 1 local NIs in recovery (showing 1): 192.168.206.102@tcp [13310.320877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13310.663690] Lustre: Found index 0 for lustre-OST0000, updating log [13311.135913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13318.111703] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13326.845549] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13330.021574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13330.038984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13335.260521] Lustre: server umount lustre-OST0000 complete [13340.448823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.456800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13340.469860] Lustre: Skipped 1 previous similar message [13340.481547] Lustre: Skipped 1 previous similar message [13342.505444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13342.518419] Lustre: Skipped 3 previous similar messages [13345.689688] Lustre: server umount lustre-MDT0000 complete [13346.601429] LustreError: 454594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13346.625942] LustreError: 454594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13349.251874] LustreError: 454575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141303 with bad export cookie 6465428370044097740 [13349.257097] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13349.261513] LustreError: 454575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13349.685433] Lustre: server umount lustre-MDT0001 complete [13369.887337] LNet: 1 local NIs in recovery (showing 1): 192.168.206.102@tcp [13379.317489] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [13388.015663] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13421.451710] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:42:52 (1789141372) [13439.273959] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [13442.184142] Key type lgssc unregistered [13442.486558] LNet: 461206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13442.500994] LNetError: 461206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13571.561331] LNet: Removed LNI 192.168.206.102@tcp [13572.686462] Key type .llcrypt unregistered [13572.688034] Key type ._llcrypt unregistered [13592.718687] Key type ._llcrypt registered [13592.725777] Key type .llcrypt registered [13592.958784] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13594.753982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13594.818689] alg: No test for adler32 (adler32-zlib) [13595.982616] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [13596.214141] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [13597.887235] Key type lgssc registered [13599.293326] Lustre: Echo OBD driver; http://www.lustre.org/ [13603.958452] LNet: Added LNI 192.168.206.102@tcp100 [8/256/0/180] [13604.874685] LNet: Added LNI 192.168.206.102@tcp101 [8/256/0/180] [13605.641144] LNet: Added LNI 192.168.206.102@tcp102 [8/256/0/180] [13606.495808] LNet: Added LNI 192.168.206.102@tcp103 [8/256/0/180] [13607.423747] LNet: Added LNI 192.168.206.102@tcp104 [8/256/0/180] [13608.293326] LNet: Added LNI 192.168.206.102@tcp105 [8/256/0/180] [13609.579239] LNet: Added LNI 192.168.206.102@tcp106 [8/256/0/180] [13610.801896] LNet: Added LNI 192.168.206.102@tcp107 [8/256/0/180] [13612.141650] LNet: Added LNI 192.168.206.102@tcp108 [8/256/0/180] [13612.961877] LNet: Added LNI 192.168.206.102@tcp109 [8/256/0/180] [13614.165297] LNet: Added LNI 192.168.206.102@tcp110 [8/256/0/180] [13615.389024] LNet: Added LNI 192.168.206.102@tcp111 [8/256/0/180] [13616.612789] LNet: Added LNI 192.168.206.102@tcp112 [8/256/0/180] [13618.124902] LNet: Added LNI 192.168.206.102@tcp113 [8/256/0/180] [13619.558215] LNet: Added LNI 192.168.206.102@tcp114 [8/256/0/180] [13621.078110] LNet: Added LNI 192.168.206.102@tcp115 [8/256/0/180] [13622.118862] LNet: Added LNI 192.168.206.102@tcp116 [8/256/0/180] [13623.247092] LNet: Added LNI 192.168.206.102@tcp117 [8/256/0/180] [13624.688251] LNet: Added LNI 192.168.206.102@tcp118 [8/256/0/180] [13625.804317] LNet: Added LNI 192.168.206.102@tcp119 [8/256/0/180] [13627.429502] LNet: Added LNI 192.168.206.102@tcp120 [8/256/0/180] [13628.703707] LNet: Added LNI 192.168.206.102@tcp121 [8/256/0/180] [13629.939671] LNet: Added LNI 192.168.206.102@tcp122 [8/256/0/180] [13631.397171] LNet: Added LNI 192.168.206.102@tcp123 [8/256/0/180] [13632.350380] LNet: Added LNI 192.168.206.102@tcp124 [8/256/0/180] [13633.747915] LNet: Added LNI 192.168.206.102@tcp125 [8/256/0/180] [13635.149632] LNet: Added LNI 192.168.206.102@tcp126 [8/256/0/180] [13636.432091] LNet: Added LNI 192.168.206.102@tcp127 [8/256/0/180] [13638.092883] LNet: Added LNI 192.168.206.102@tcp128 [8/256/0/180] [13639.247150] LNet: Added LNI 192.168.206.102@tcp129 [8/256/0/180] [13640.618905] LNet: Added LNI 192.168.206.102@tcp130 [8/256/0/180] [13641.603575] LNet: Added LNI 192.168.206.102@tcp131 [8/256/0/180] [13642.434492] LNet: Added LNI 192.168.206.102@tcp132 [8/256/0/180] [13643.285433] LNet: Added LNI 192.168.206.102@tcp133 [8/256/0/180] [13644.267265] LNet: Added LNI 192.168.206.102@tcp134 [8/256/0/180] [13645.306086] LNet: Added LNI 192.168.206.102@tcp135 [8/256/0/180] [13652.746914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13652.799757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13653.944654] Lustre: 464254:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13654.037949] LNetError: 464274:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13654.061616] LNetError: Protocol error connecting to 192.168.206.102@tcp121 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13654.075151] LNetError: Skipped 4 previous similar messages [13654.191867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13654.235594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13654.358342] Lustre: lustre-MDT0000: new disk, initializing [13654.479143] Lustre: 464309:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13654.495786] Lustre: 464309:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13654.533541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13654.556962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13654.629742] LNet: 7 local NIs in recovery (showing 7): 192.168.206.102@tcp122, 192.168.206.102@tcp125, 192.168.206.102@tcp121, 192.168.206.102@tcp128, 192.168.206.102@tcp126, 192.168.206.102@tcp127, 192.168.206.102@tcp129 [13655.148559] LNetError: 464280:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13655.167815] LNetError: 464280:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [13655.182182] LNetError: Protocol error connecting to 192.168.206.102@tcp114 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13655.198611] LNetError: Skipped 7 previous similar messages [13657.761278] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13657.788980] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13657.804583] LNetError: Protocol error connecting to 192.168.206.102@tcp122 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13657.818671] LNetError: Skipped 8 previous similar messages [13659.959809] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13661.857483] LNetError: 464275:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13661.877331] LNetError: 464275:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [13661.896946] LNetError: Protocol error connecting to 192.168.206.102@tcp122 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13669.025904] LNetError: 464280:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13669.048498] LNetError: Protocol error connecting to 192.168.206.102@tcp122 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13670.737959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13670.946844] Lustre: 464269:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13671.020816] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13671.027209] Lustre: Skipped 1 previous similar message [13671.088583] Lustre: lustre-MDT0001: new disk, initializing [13671.114796] Lustre: 465071:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13671.124210] Lustre: 465071:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13671.250396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13671.290641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13671.307674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13676.008090] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13680.719553] Lustre: 465496:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13680.729485] Lustre: 465496:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13685.426184] LNetError: 464274:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13685.434897] LNetError: Protocol error connecting to 192.168.206.102@tcp122 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13685.768659] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13691.822455] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13701.072128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13701.427658] Lustre: lustre-OST0000: new disk, initializing [13701.459724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13701.466385] Lustre: 466244:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13701.557466] Lustre: 466249:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13701.562827] Lustre: 466249:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13701.618244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13705.761075] LNetError: 461713:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13705.773498] LNetError: 461713:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13705.790454] LNetError: Protocol error connecting to 192.168.206.102@tcp119 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13705.812730] LNetError: Skipped 8 previous similar messages [13708.110843] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13710.880971] Lustre: 466700:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.102@tcp: can't add NID 192.168.206.102@tcp130: rc = -75 [13710.896425] Lustre: 466700:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13710.969953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13710.979170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13711.135712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13715.104422] LNet: 1 local NIs in recovery (showing 1): 192.168.206.102@tcp122 [13717.767167] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13726.697946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13726.702128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13726.703298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13726.720100] Lustre: Skipped 1 previous similar message [13731.161289] Lustre: server umount lustre-OST0000 complete [13736.928458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.929606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13736.959775] Lustre: Skipped 1 previous similar message [13736.966513] Lustre: Skipped 1 previous similar message [13741.496934] Lustre: server umount lustre-MDT0000 complete [13745.698314] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.102 [13745.722523] LNetError: 464277:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 15 previous similar messages [13745.741921] LNetError: Protocol error connecting to 192.168.206.102@tcp114 at host 192.168.206.102:988: is it running a compatible version of Lustre? [13745.771188] LNetError: Skipped 15 previous similar messages [13745.883249] LustreError: 464264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789141699 with bad export cookie 12705118920085150316 [13745.889911] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13745.893560] LustreError: 464264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13746.329106] Lustre: server umount lustre-MDT0001 complete [13757.504133] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [13760.676671] Key type lgssc unregistered [13761.140647] LNet: 468160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13761.147587] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13761.154965] LNet: Removed LNI 192.168.206.102@tcp [13762.207695] LNet: Removed LNI 192.168.206.102@tcp100 [13762.236313] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13762.255433] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13762.269876] LNet: Removed LNI 192.168.206.102@tcp101 [13763.299780] LNet: Removed LNI 192.168.206.102@tcp102 [13763.321480] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13763.330688] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13763.338903] LNet: Removed LNI 192.168.206.102@tcp103 [13764.383220] LNet: Removed LNI 192.168.206.102@tcp104 [13765.408824] LNet: Removed LNI 192.168.206.102@tcp105 [13765.411118] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13765.424441] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13765.430674] LNet: Removed LNI 192.168.206.102@tcp106 [13766.496493] LNet: Removed LNI 192.168.206.102@tcp107 [13767.519305] LNet: Removed LNI 192.168.206.102@tcp108 [13768.543893] LNet: Removed LNI 192.168.206.102@tcp109 [13769.569562] LNet: Removed LNI 192.168.206.102@tcp110 [13769.576559] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13769.584098] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13769.593117] LNet: Removed LNI 192.168.206.102@tcp111 [13770.655157] LNet: Removed LNI 192.168.206.102@tcp112 [13771.679646] LNet: Removed LNI 192.168.206.102@tcp113 [13772.704639] LNet: Removed LNI 192.168.206.102@tcp114 [13773.727175] LNet: Removed LNI 192.168.206.102@tcp115 [13774.751201] LNet: Removed LNI 192.168.206.102@tcp116 [13775.775176] LNet: Removed LNI 192.168.206.102@tcp117 [13776.799220] LNet: Removed LNI 192.168.206.102@tcp118 [13777.823177] LNet: Removed LNI 192.168.206.102@tcp119 [13777.827266] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13777.841239] LNetError: 468160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13777.850871] LNet: Removed LNI 192.168.206.102@tcp120 [13778.911316] LNet: Removed LNI 192.168.206.102@tcp121 [13778.912995] LNet: Removed LNI 192.168.206.102@tcp122 [13779.935191] LNet: Removed LNI 192.168.206.102@tcp123 [13780.959190] LNet: Removed LNI 192.168.206.102@tcp124 [13781.983160] LNet: Removed LNI 192.168.206.102@tcp125 [13783.007415] LNet: Removed LNI 192.168.206.102@tcp126 [13784.031527] LNet: Removed LNI 192.168.206.102@tcp127 [13785.055212] LNet: Removed LNI 192.168.206.102@tcp128 [13786.079375] LNet: Removed LNI 192.168.206.102@tcp129 [13786.081478] LNet: Removed LNI 192.168.206.102@tcp130 [13786.098680] LNet: Removed LNI 192.168.206.102@tcp131 [13786.102273] LNet: Removed LNI 192.168.206.102@tcp132 [13786.119097] LNet: Removed LNI 192.168.206.102@tcp133 [13786.122562] LNet: Removed LNI 192.168.206.102@tcp134 [13786.133241] LNet: Removed LNI 192.168.206.102@tcp135 [13787.221776] Key type .llcrypt unregistered [13787.224954] Key type ._llcrypt unregistered [13811.915233] Key type ._llcrypt registered [13811.917313] Key type .llcrypt registered [13812.051527] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [13828.437437] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13829.639904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13829.751400] alg: No test for adler32 (adler32-zlib) [13830.959844] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [13831.272709] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [13832.919225] Key type lgssc registered [13834.086538] Lustre: Echo OBD driver; http://www.lustre.org/ [13869.941812] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:50:21 (1789141821) [13888.744710] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [13891.641678] Key type lgssc unregistered [13891.927747] LNet: 472252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13891.938958] LNetError: 472252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13891.964201] LNet: Removed LNI 192.168.206.102@tcp [13892.698330] Key type .llcrypt unregistered [13892.700864] Key type ._llcrypt unregistered [13909.088176] Key type ._llcrypt registered [13909.092098] Key type .llcrypt registered [13909.202283] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [13910.025638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13910.107258] alg: No test for adler32 (adler32-zlib) [13911.186926] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [13911.487330] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [13913.159932] Key type lgssc registered [13914.229834] Lustre: Echo OBD driver; http://www.lustre.org/ [13924.959551] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13924.995167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13926.354984] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13926.385759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13926.498232] Lustre: lustre-MDT0000: new disk, initializing [13926.656552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13926.678345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13932.427684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13944.646935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13944.862492] Lustre: 473465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13944.916339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13944.929931] Lustre: Skipped 1 previous similar message [13945.002378] Lustre: lustre-MDT0001: new disk, initializing [13945.145430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13945.236966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13945.290235] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13950.068217] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13960.015658] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13966.112770] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13972.921821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13973.259890] Lustre: lustre-OST0000: new disk, initializing [13973.263489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13973.271885] Lustre: 475438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13973.369671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13979.562602] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [13979.770422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13979.791705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13979.940420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13990.130811] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13995.101557] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing lnet_if_list [14001.837339] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14001.897944] LNet: Added LNI 192.168.206.102@tcp10 [8/256/0/180] [14009.263765] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14009.294370] LNet: Added LNI 192.168.206.102@tcp11 [8/256/0/180] [14013.711543] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14013.736860] LNet: Added LNI 192.168.206.102@tcp12 [8/256/0/180] [14018.275285] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14018.354047] LNet: Added LNI 192.168.206.102@tcp13 [8/256/0/180] [14022.962814] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14023.001581] LNet: Added LNI 192.168.206.102@tcp14 [8/256/0/180] [14027.618650] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14027.643616] LNet: Added LNI 192.168.206.102@tcp15 [8/256/0/180] [14032.841785] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14032.884960] LNet: Added LNI 192.168.206.102@tcp16 [8/256/0/180] [14037.572549] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14037.601513] LNet: Added LNI 192.168.206.102@tcp17 [8/256/0/180] [14042.561843] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14042.598215] LNet: Added LNI 192.168.206.102@tcp18 [8/256/0/180] [14046.879542] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14046.909526] LNet: Added LNI 192.168.206.102@tcp19 [8/256/0/180] [14051.284637] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14051.332270] LNet: Added LNI 192.168.206.102@tcp20 [8/256/0/180] [14056.016897] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14056.051258] LNet: Added LNI 192.168.206.102@tcp21 [8/256/0/180] [14061.139201] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14061.190879] LNet: Added LNI 192.168.206.102@tcp22 [8/256/0/180] [14065.994700] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14066.031320] LNet: Added LNI 192.168.206.102@tcp23 [8/256/0/180] [14071.456641] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14071.487211] LNet: Added LNI 192.168.206.102@tcp24 [8/256/0/180] [14076.611513] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14076.658381] LNet: Added LNI 192.168.206.102@tcp25 [8/256/0/180] [14082.710363] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14082.730386] LNet: Added LNI 192.168.206.102@tcp26 [8/256/0/180] [14088.701946] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14088.727698] LNet: Added LNI 192.168.206.102@tcp27 [8/256/0/180] [14094.400311] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14094.427850] LNet: Added LNI 192.168.206.102@tcp28 [8/256/0/180] [14100.597552] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14100.628084] LNet: Added LNI 192.168.206.102@tcp29 [8/256/0/180] [14107.323780] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14107.357967] LNet: Added LNI 192.168.206.102@tcp30 [8/256/0/180] [14114.147220] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14114.194581] LNet: Added LNI 192.168.206.102@tcp31 [8/256/0/180] [14121.265831] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14121.293400] LNet: Added LNI 192.168.206.102@tcp32 [8/256/0/180] [14127.704988] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14127.734747] LNet: Added LNI 192.168.206.102@tcp33 [8/256/0/180] [14134.893929] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14134.927662] LNet: Added LNI 192.168.206.102@tcp34 [8/256/0/180] [14142.847230] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14142.883052] LNet: Added LNI 192.168.206.102@tcp35 [8/256/0/180] [14150.368624] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14150.394740] LNet: Added LNI 192.168.206.102@tcp36 [8/256/0/180] [14160.778390] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14160.818711] LNet: Added LNI 192.168.206.102@tcp37 [8/256/0/180] [14167.536313] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14167.574753] LNet: Added LNI 192.168.206.102@tcp38 [8/256/0/180] [14175.458359] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14175.490342] LNet: Added LNI 192.168.206.102@tcp39 [8/256/0/180] [14183.525983] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14183.567814] LNet: Added LNI 192.168.206.102@tcp40 [8/256/0/180] [14193.584487] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14193.611374] LNet: Added LNI 192.168.206.102@tcp41 [8/256/0/180] [14201.318715] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14201.354371] LNet: Added LNI 192.168.206.102@tcp42 [8/256/0/180] [14209.887516] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14209.941951] LNet: Added LNI 192.168.206.102@tcp43 [8/256/0/180] [14218.158969] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14218.205645] LNet: Added LNI 192.168.206.102@tcp44 [8/256/0/180] [14229.738647] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14229.814441] LNet: Added LNI 192.168.206.102@tcp45 [8/256/0/180] [14241.349684] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14241.392733] LNet: Added LNI 192.168.206.102@tcp46 [8/256/0/180] [14250.809337] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14250.833906] LNet: Added LNI 192.168.206.102@tcp47 [8/256/0/180] [14258.846808] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14258.877209] LNet: Added LNI 192.168.206.102@tcp48 [8/256/0/180] [14267.948045] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14267.981152] LNet: Added LNI 192.168.206.102@tcp49 [8/256/0/180] [14276.692397] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14276.734884] LNet: Added LNI 192.168.206.102@tcp50 [8/256/0/180] [14302.688883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14302.690028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14302.698704] Lustre: Skipped 1 previous similar message [14302.714558] Lustre: Skipped 2 previous similar messages [14306.435838] Lustre: server umount lustre-MDT0000 complete [14310.807291] LustreError: 473458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142264 with bad export cookie 9283352663740887218 [14310.813019] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14310.823211] LustreError: 473458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14311.313305] Lustre: server umount lustre-MDT0001 complete [14328.291146] Lustre: 472739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789142266/real 1789142266] req@000000009da4c590 x1876051220411008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789142282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14328.331638] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14328.344732] Lustre: Skipped 1 previous similar message [14332.192106] Lustre: 472738:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789142269/real 1789142269] req@0000000059d5afe5 x1876051220411136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789142285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14332.308110] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [14333.423584] Lustre: 472740:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789142271/real 1789142271] req@00000000b20fe351 x1876051220411264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789142287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14336.481570] Lustre: 472741:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789142274/real 1789142274] req@00000000511afb62 x1876051220411520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789142290 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14342.465479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14342.624994] Lustre: 472739:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789142280/real 1789142280] req@00000000c29ca589 x1876051220411648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789142296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14356.000579] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x80d5180ff6f3bad0 [14356.011725] Lustre: MGC192.168.206.102@tcp: Connection restored to 0@lo (at 0@lo) [14356.258366] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14356.274956] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [14356.324455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14357.345387] LustreError: 475430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14357.351043] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14358.368137] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14360.836887] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14363.621179] LustreError: 475430:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14368.626164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.740493] LustreError: 475429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14369.053680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14373.734368] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14374.207596] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14380.762840] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14386.166155] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14405.091918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14405.093266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14405.106422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14405.113915] Lustre: Skipped 1 previous similar message [14410.218977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14410.228861] Lustre: Skipped 1 previous similar message [14415.340149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14415.351716] Lustre: Skipped 1 previous similar message [14416.863171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14417.006776] Lustre: server umount lustre-OST0000 complete [14420.460717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14420.462814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14420.463782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14420.498425] Lustre: Skipped 2 previous similar messages [14426.469792] Lustre: server umount lustre-MDT0000 complete [14430.575494] LustreError: 475436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142384 with bad export cookie 9283352663740889808 [14430.589507] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14430.593398] LustreError: 475436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14431.064654] Lustre: server umount lustre-MDT0001 complete [14443.624489] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [14446.216063] Key type lgssc unregistered [14446.605601] LNet: 502490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14446.613122] LNetError: 502490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14446.620467] LNet: Removed LNI 192.168.206.102@tcp [14446.623768] LNet: Removed LNI 192.168.206.102@tcp10 [14446.627496] LNet: Removed LNI 192.168.206.102@tcp11 [14446.630389] LNet: Removed LNI 192.168.206.102@tcp12 [14446.632995] LNet: Removed LNI 192.168.206.102@tcp13 [14446.636551] LNet: Removed LNI 192.168.206.102@tcp14 [14446.640257] LNet: Removed LNI 192.168.206.102@tcp15 [14446.644409] LNet: Removed LNI 192.168.206.102@tcp16 [14446.647689] LNet: Removed LNI 192.168.206.102@tcp17 [14446.650696] LNet: Removed LNI 192.168.206.102@tcp18 [14446.653384] LNet: Removed LNI 192.168.206.102@tcp19 [14446.656356] LNet: Removed LNI 192.168.206.102@tcp20 [14446.658788] LNet: Removed LNI 192.168.206.102@tcp21 [14446.660801] LNet: Removed LNI 192.168.206.102@tcp22 [14446.663735] LNet: Removed LNI 192.168.206.102@tcp23 [14446.666576] LNet: Removed LNI 192.168.206.102@tcp24 [14446.669277] LNet: Removed LNI 192.168.206.102@tcp25 [14446.671925] LNet: Removed LNI 192.168.206.102@tcp26 [14446.674902] LNet: Removed LNI 192.168.206.102@tcp27 [14446.677586] LNet: Removed LNI 192.168.206.102@tcp28 [14446.680716] LNet: Removed LNI 192.168.206.102@tcp29 [14446.684511] LNet: Removed LNI 192.168.206.102@tcp30 [14446.688527] LNet: Removed LNI 192.168.206.102@tcp31 [14446.691979] LNet: Removed LNI 192.168.206.102@tcp32 [14446.695272] LNet: Removed LNI 192.168.206.102@tcp33 [14446.699437] LNet: Removed LNI 192.168.206.102@tcp34 [14446.705318] LNet: Removed LNI 192.168.206.102@tcp35 [14446.711504] LNet: Removed LNI 192.168.206.102@tcp36 [14446.723068] LNet: Removed LNI 192.168.206.102@tcp37 [14446.727708] LNet: Removed LNI 192.168.206.102@tcp38 [14446.735031] LNet: Removed LNI 192.168.206.102@tcp39 [14446.741919] LNet: Removed LNI 192.168.206.102@tcp40 [14446.748470] LNet: Removed LNI 192.168.206.102@tcp41 [14446.751174] LNet: Removed LNI 192.168.206.102@tcp42 [14446.753908] LNet: Removed LNI 192.168.206.102@tcp43 [14446.758236] LNet: Removed LNI 192.168.206.102@tcp44 [14446.764705] LNet: Removed LNI 192.168.206.102@tcp45 [14446.767656] LNet: Removed LNI 192.168.206.102@tcp46 [14446.771823] LNet: Removed LNI 192.168.206.102@tcp47 [14446.773483] LNet: Removed LNI 192.168.206.102@tcp48 [14446.779270] LNet: Removed LNI 192.168.206.102@tcp49 [14446.788063] LNet: Removed LNI 192.168.206.102@tcp50 [14447.599922] Key type .llcrypt unregistered [14447.606371] Key type ._llcrypt unregistered [14466.664100] Key type ._llcrypt registered [14466.667901] Key type .llcrypt registered [14466.772491] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [14479.851156] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [14480.944232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14480.953927] alg: No test for adler32 (adler32-zlib) [14482.082713] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [14482.367526] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [14484.087315] Key type lgssc registered [14485.766667] Lustre: Echo OBD driver; http://www.lustre.org/ [14523.270761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14523.290505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14524.478932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14524.525588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14524.610590] Lustre: lustre-MDT0000: new disk, initializing [14524.669329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14524.685679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14530.327891] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14541.525640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14541.658227] Lustre: 505677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14541.726490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14541.731923] Lustre: Skipped 1 previous similar message [14541.836562] Lustre: lustre-MDT0001: new disk, initializing [14541.978631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14542.035578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14542.053975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14546.970422] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14558.039995] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14564.805761] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14572.704229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14572.880391] Lustre: *** cfs_fail_loc=90f, val=0*** [14572.945945] Lustre: lustre-OST0000: new disk, initializing [14572.950784] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14572.957476] Lustre: 507693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14573.084744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14579.141887] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14582.430088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14582.447912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14582.575363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14589.202956] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14598.122324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14598.123667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14598.136299] LustreError: Skipped 1 previous similar message [14598.137409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14598.173661] Lustre: Skipped 1 previous similar message [14601.730864] Lustre: server umount lustre-OST0000 complete [14608.356131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14608.360093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14608.371594] Lustre: Skipped 1 previous similar message [14608.388705] Lustre: Skipped 2 previous similar messages [14611.831763] Lustre: server umount lustre-MDT0000 complete [14615.679101] LustreError: 505670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142569 with bad export cookie 17337947273607004502 [14615.690160] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14615.706763] LustreError: 505670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14616.273437] Lustre: server umount lustre-MDT0001 complete [14627.367418] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [14630.196615] Key type lgssc unregistered [14630.523851] LNet: 509603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14630.535454] LNetError: 509603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14631.597576] LNet: Removed LNI 192.168.206.102@tcp [14632.510707] Key type .llcrypt unregistered [14632.515435] Key type ._llcrypt unregistered [14651.230746] Key type ._llcrypt registered [14651.236605] Key type .llcrypt registered [14651.353340] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [14652.356258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14652.542615] alg: No test for adler32 (adler32-zlib) [14653.624622] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [14653.856243] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [14655.503335] Key type lgssc registered [14656.715989] Lustre: Echo OBD driver; http://www.lustre.org/ [14663.304817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14663.319489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14664.715351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14669.117276] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14674.914974] LustreError: 510868:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14674.953619] LustreError: 510868:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [14678.036025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14678.385727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14682.544423] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14690.265183] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14696.480883] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14703.038816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14703.491457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14710.597154] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14718.618245] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14729.491232] loop0: detected capacity change from 0 to 204800000 [14729.635582] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14729.654191] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14729.665220] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14736.399640] loop0: detected capacity change from 0 to 204800000 [14736.662457] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14736.680695] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14736.697767] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14742.444868] loop0: detected capacity change from 0 to 204800000 [14748.575793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14748.736776] Lustre: fs15246-OST0000: new disk, initializing [14748.742301] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14748.750428] Lustre: 514463:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14748.836437] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14755.726805] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14766.102954] loop1: detected capacity change from 0 to 204800000 [14772.072657] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14772.199275] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14772.256836] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14772.332711] Lustre: fs15246-MDT0000: new disk, initializing [14772.544361] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14772.577192] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14772.696958] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14773.728679] Lustre: 515589:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000011a79d3f x1876051998323840/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789142743 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14777.355618] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14782.896560] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14782.914681] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14782.955991] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14801.898662] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14801.921411] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14804.330384] Lustre: server umount fs15246-MDT0000 complete [14807.017898] LustreError: 510862:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14812.139233] LustreError: 515253:0:(ldlm_lib.c:1228:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14813.841652] Lustre: server umount fs15246-OST0000 complete [14827.491397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14827.496724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14827.510527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14827.523577] LustreError: Skipped 1 previous similar message [14827.547676] Lustre: Skipped 1 previous similar message [14831.593684] Lustre: server umount lustre-OST0000 complete [14837.217176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14837.230143] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14837.248657] Lustre: Skipped 1 previous similar message [14837.259074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14841.707199] Lustre: server umount lustre-MDT0000 complete [14842.872237] LustreError: 511908:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14842.913607] LustreError: 511908:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [14845.422799] LustreError: 510850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789142799 with bad export cookie 3869545589452354073 [14845.435089] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14845.436685] LustreError: 510850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14845.699553] Lustre: server umount lustre-MDT0001 complete [14856.747372] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [14860.167404] Key type lgssc unregistered [14860.512868] LNet: 518687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14860.527477] LNetError: 518687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14860.551308] LNet: Removed LNI 192.168.206.102@tcp [14861.428830] Key type .llcrypt unregistered [14861.434238] Key type ._llcrypt unregistered [14881.918075] Key type ._llcrypt registered [14881.922148] Key type .llcrypt registered [14882.038491] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [14884.073841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14884.152562] alg: No test for adler32 (adler32-zlib) [14885.331970] Lustre: Lustre: Build Version: 2.17.57_135_g8edbab4 [14885.639662] LNet: Added LNI 192.168.206.102@tcp [8/256/0/180] [14887.343923] Key type lgssc registered [14888.893928] Lustre: Echo OBD driver; http://www.lustre.org/ [14931.922874] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [14942.562330] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [14979.897521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14979.920170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14981.221926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14981.264387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14981.360719] Lustre: lustre-MDT0000: new disk, initializing [14981.450508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14981.465664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14985.222494] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [14996.390421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14996.666791] Lustre: 523158:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14996.765027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14996.770575] Lustre: Skipped 1 previous similar message [14996.943144] Lustre: lustre-MDT0001: new disk, initializing [14997.158697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14997.259037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14997.270641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15002.378800] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [15012.665907] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15018.986242] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15026.316831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15026.543237] Lustre: lustre-OST0000: new disk, initializing [15026.547477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15026.553922] Lustre: 525126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15026.644212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15033.950151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15033.962777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15034.052167] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_default_debug -1 all [15034.102402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15044.375380] Lustre: DEBUG MARKER: oleg602-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15049.645215] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15049.890884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15054.719363] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15054.945346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15059.943345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15059.944104] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15059.949342] LustreError: Skipped 1 previous similar message [15059.952956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15059.965881] Lustre: Skipped 1 previous similar message [15063.560204] Lustre: server umount lustre-OST0000 complete [15068.641242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15068.658065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15068.673159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15068.681239] Lustre: Skipped 1 previous similar message [15070.185141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15070.205273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15074.513826] Lustre: server umount lustre-MDT0000 complete [15078.438921] LustreError: 523149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789143032 with bad export cookie 14232436051959325233 [15078.451957] LustreError: 523149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15078.452933] LustreError: MGC192.168.206.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15078.837514] Lustre: server umount lustre-MDT0001 complete [15105.283065] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing set_hostid [15116.870894] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing load_modules_local [15147.421200] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14896 sec ========= 12:11:38 (1789143098) [15149.666931] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:11:40 (1789143100) === [15154.308627] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:11:45 (1789143105) === [15197.378941] Lustre: DEBUG MARKER: oleg602-server.virtnet: executing unload_modules_local [15200.278411] Key type lgssc unregistered [15200.555605] LNet: 531777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15200.558923] LNetError: 531777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15200.568806] LNet: Removed LNI 192.168.206.102@tcp [15201.366582] Key type .llcrypt unregistered [15201.370124] Key type ._llcrypt unregistered